From: DRISCOLL, ROBERT (ROBDRIIZZATSAFECO.com)
Date: Fri Aug 27 2004 - 22:14:53 EDT

  • Next message: Steve Zohn: "[load balancing] load balancing firewalls"
    RE: [load balancing] F5 BigIP out of memory: SSL proxies failed - sites down

    I saw this on 4.5ptf7 on the F5 E-Commerce Controllers as well. The 2 processor systems had an occurance of proxyd for each proc. Eventually the free memory would shrink and each proxyd would continue to hold a large sum of memory. When the systems got busy it would consume more memory, but when the load started to fall off, proxyd would not give the memory back. Killing proxyd and having checkd restart it would temporarily solve the problem.

    Since we upgraded to 4.5ptf10 the issue has gone away.


    -----Original Message-----
    From:   owner-lb-lIZZATvegan.net on behalf of Fletcher Cocquyt
    Sent:   Fri 8/27/2004 2:36 PM
    To:     lb-lIZZATvegan.net
    Cc:    
    Subject:        RE: [load balancing] F5 BigIP out of memory: SSL proxies failed - sites down
    Sorry, I SNMP reports the wrong PTF level - this is actually 4.2 PTF-6
    I will look at PTF10 - but thought I'd ask if anyone else has seen this
    - what appears to be perhaps a memory leak condition?

    Thanks,

    Fletcher Cocquyt
    Senior Systems Administrator
    fcocquytIZZATstanford.edu

    -----Original Message-----
    From: owner-lb-lIZZATvegan.net [mailto:owner-lb-lIZZATvegan.net] On Behalf Of
    Fletcher Cocquyt
    Sent: Friday, August 27, 2004 10:34 AM
    To: lb-lIZZATvegan.net
    Subject: [load balancing] F5 BigIP out of memory: SSL proxies failed -
    sites down

    Fellow lb'ers,

    Ran into an out of memory condition on the active BigIP which ended up
    in the SSL proxied sites eventually becoming unreachable.

    These messages preceeded the out of memory condition:
    From bigip log:

    Aug 26 12:52:36 irt-lb-02 kernel: ssl: save_parsed_ssl_persist: invalid
    packet from IP1:2581 to VIP1:443
    Aug 26 12:52:59 irt-lb-02 kernel: SSL3: Content type not a server hello.
    Aug 26 12:52:59 irt-lb-02 kernel: ssl: save_parsed_ssl_persist: invalid
    packet from IP1:2582 to VIP1:443
    Aug 26 13:25:00 irt-lb-02 kernel: SSL3: Content type not a server hello.
    Aug 26 13:25:00 irt-lb-02 kernel: ssl: save_parsed_ssl_persist: invalid
    packet from IP2:1368 to VIP2:443
    Aug 26 15:03:55 irt-lb-02 kernel: bigmsg2_switch: out of memory
    Aug 26 15:10:01 irt-lb-02 kernel: bigmsg2_switch: out of memory
    Aug 26 15:13:51 irt-lb-02 kernel: bigmsg2_switch: out of memory
    Aug 26 15:15:42 irt-lb-02 kernel: SSL3: Content type not a server hello.
    Aug 26 15:15:42 irt-lb-02 kernel: ssl: save_parsed_ssl_persist: invalid
    packet from IP3:2615 to VIP1:443
    Aug 26 15:16:53 irt-lb-02 kernel: bigmsg2_switch: out of memory
    Aug 26 15:18:05 irt-lb-02 kernel: bigmsg2_switch: out of memory
    Aug 26 15:20:03 irt-lb-02 kernel: bigmsg2_switch: out of memory

    And corresponding entries from bigd log:

    Aug 26 15:10:01 irt-lb-02 checkd: No memory available for operation
    Aug 26 15:13:51 irt-lb-02 checkd: No memory available for operation
    Aug 26 15:16:53 irt-lb-02 checkd: No memory available for operation

    I ended up failing over manually to bring the sites back.

    This is a BIG-IP 4.2PTF-0 (I am looking at the patch lists now)

    Has anyone seen this before?
    What can be done to detect and prevent a site down situation as this
    became?

    Thanks,

    Fletcher Cocquyt
    Senior Systems Administrator
    fcocquytIZZATstanford.edu



    ____________________ The Load Balancing Mailing List Unsubscribe: mailto:majordomoIZZATvegan.net?body=unsubscribe%20lb-l Archive: http://vegan.net/lb/archive LBDigest: http://lbdigest.com MRTG with SLB: http://vegan.net/MRTG Hosted by: http://www.tokkisystems.com

    This archive was generated by hypermail 2.1.4 : Fri Aug 27 2004 - 22:24:59 EDT