Uploaded image for project: 'Snowstorm'
  1. Snowstorm
  2. STORM-1122

Linux viewer sucks up file descriptors, stops loading content and crashes

    XMLWordPrintableJSON

    Details

    • Type: Defect
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Bug Queue: viewer-development
    • Labels:
    • Environment:
    • Test Plan:
      Hide

      1. Log in
      2. Examine file descriptor count with `lsof -c do-not | wc -l`
      3. Ensure count is far below 1024 (ideally but not necessarily less than 256 as the default may be low on some systems)
      4. Observe that only DejaVu fonts included with the viewer are used by checking their counts with `lsof -c do-not | egrep -o '[^ ]font[^ ]' | sort | uniq -c | less` (some shared libs may show up due to the grep pattern).

      Show
      1. Log in 2. Examine file descriptor count with `lsof -c do-not | wc -l` 3. Ensure count is far below 1024 (ideally but not necessarily less than 256 as the default may be low on some systems) 4. Observe that only DejaVu fonts included with the viewer are used by checking their counts with `lsof -c do-not | egrep -o ' [^ ] font [^ ] ' | sort | uniq -c | less` (some shared libs may show up due to the grep pattern).

      Description

      Observed behavior:
      Viewer maintains >900 font file descriptors (less or more depending on your system, apparently); instability results, file descriptor usage counts look like this immediately after logging in (from build 2.6.0.225006 as well as dev):
      20 /home/n/bin/SecondLife-i686-2.6.0.225006/fonts/DejaVuSans-Bold.ttf
      1 /home/n/bin/SecondLife-i686-2.6.0.225006/fonts/DejaVuSansMono.ttf
      25 /home/n/bin/SecondLife-i686-2.6.0.225006/fonts/DejaVuSans.ttf
      1 /home/n/.fontconfig/027be39b16fb8096c5520a8560195d8f-le32d4.cache-3
      1 /home/n/.fontconfig/062808c12e6e608270f93bb230aed730-le32d4.cache-3
      1 /home/n/.fontconfig/089dead882dea3570ffc31a9898cfb69-le32d4.cache-3
      1 /home/n/.fontconfig/0d8c3b2ac0904cb8a57a757ad11a4a08-le32d4.cache-3
      1 /home/n/.fontconfig/0dad82dbaa6c15cf0806f139d62298a3-le32d4.cache-3
      1 /home/n/.fontconfig/0f34bcd4b6ee430af32735b75db7f02b-le32d4.cache-3
      1 /home/n/.fontconfig/198d8fcf01c96d0cf813f74fd759bdb7-le32d4.cache-3
      1 /home/n/.fontconfig/21e44d52fe27b05380d683e98a23dc93-le32d4.cache-3
      1 /home/n/.fontconfig/2cd17615ca594fa2959ae173292e504c-le32d4.cache-3
      1 /home/n/.fontconfig/30aa24756b457425d399a9ac9e5092fe-le32d4.cache-3
      1 /home/n/.fontconfig/365b55f210c0a22e9a19e35191240f32-le32d4.cache-3
      1 /home/n/.fontconfig/37eaa8989bf5ced22c1cafa0a4fbfbc9-le32d4.cache-3
      1 /home/n/.fontconfig/401a5dd6b567794a1d18dd9342dfa604-le32d4.cache-3
      1 /home/n/.fontconfig/406bd5c19e5cc517440ee75488dad48e-le32d4.cache-3
      1 /home/n/.fontconfig/4345ce61bb1f6efbc6be7b9f7dc82408-le32d4.cache-3
      1 /home/n/.fontconfig/44492eb4b9ff25a2942da984b73bd713-le32d4.cache-3
      1 /home/n/.fontconfig/468ad6f9f7b5a47b6ca8a3bfb2d83cf9-le32d4.cache-3
      1 /home/n/.fontconfig/4794a0821666d79190d59a36cb4f44b5-le32d4.cache-3
      1 /home/n/.fontconfig/4c73fe0c47614734b17d736dbde7580a-le32d4.cache-3
      1 /home/n/.fontconfig/4c8bd476623eafb6fe862291a3b05f7d-le32d4.cache-3
      1 /home/n/.fontconfig/515ca1ebc4b18308bea979be5704f9db-le32d4.cache-3
      1 /home/n/.fontconfig/524d0bf16b95950b2446ce127a664042-le32d4.cache-3
      1 /home/n/.fontconfig/5a76b20910bc01daa5b6abd117531ab8-le32d4.cache-3
      1 /home/n/.fontconfig/5aa9259560595826861fba5056bf4850-le32d4.cache-3
      1 /home/n/.fontconfig/5b3d50f66ac7b6dadb07df5088d0f9d2-le32d4.cache-3
      1 /home/n/.fontconfig/6a53c69dea097a2d716e069445527da8-le32d4.cache-3
      1 /home/n/.fontconfig/6d41288fd70b0be22e8c3a91e032eec0-le32d4.cache-3
      1 /home/n/.fontconfig/6eb3985aa4124903f6ff08ba781cd364-le32d4.cache-3
      1 /home/n/.fontconfig/6fe0ed9ceabbaa79283ea9c0b49c5d6e-le32d4.cache-3
      1 /home/n/.fontconfig/79b7902a698c37d747b157374a08587f-le32d4.cache-3
      1 /home/n/.fontconfig/7c5329225a7a79882b13f39a3022c7b0-le32d4.cache-3
      1 /home/n/.fontconfig/7e34cecb860548a1af81efa9b74e94a9-le32d4.cache-3
      1 /home/n/.fontconfig/7ee55724f82591cb35c3d9771e9e69ed-le32d4.cache-3
      1 /home/n/.fontconfig/7ef2298fde41cc6eeb7af42e48b7d293-le32d4.cache-3
      1 /home/n/.fontconfig/80d0a1cf7989637baed91f874e3e11f6-le32d4.cache-3
      1 /home/n/.fontconfig/85130c034ee6c6a57445579585c0b546-le32d4.cache-3
      1 /home/n/.fontconfig/865f88548240fee46819705c6468c165-le32d4.cache-3
      1 /home/n/.fontconfig/8fb5441df18e7cfa96dc4c83aed09ee1-le32d4.cache-3
      1 /home/n/.fontconfig/926e794c3d5e5dffcaf2fa83ef8d36c2-le32d4.cache-3
      1 /home/n/.fontconfig/945677eb7aeaf62f1d50efc3fb3ec7d8-le32d4.cache-3
      1 /home/n/.fontconfig/946752ae7a90c323083f887d43ff0bb2-le32d4.cache-3
      1 /home/n/.fontconfig/9923581aef9bb9211ee5bcd557562ee7-le32d4.cache-3
      1 /home/n/.fontconfig/99e8ed0e538f840c565b6ed5dad60d56-le32d4.cache-3
      1 /home/n/.fontconfig/9a04f3458e11652314d3177aded2a140-le32d4.cache-3
      1 /home/n/.fontconfig/a755afe4a08bf5b97852ceb7400b47bc-le32d4.cache-3
      1 /home/n/.fontconfig/a8b5dbdd3d19cfe4c974afde17f13673-le32d4.cache-3
      1 /home/n/.fontconfig/b595078a2751c5bfe0e263251cd3c6c2-le32d4.cache-3
      1 /home/n/.fontconfig/b5ea634b0fb353b8ea17632d1f9ef766-le32d4.cache-3
      1 /home/n/.fontconfig/b648d122f164df55c519c0028a02f0c4-le32d4.cache-3
      1 /home/n/.fontconfig/c05880de57d1f5e948fdfacc138775d9-le32d4.cache-3
      1 /home/n/.fontconfig/c28e2896718661e2d285ba74f818ce43-le32d4.cache-3
      1 /home/n/.fontconfig/cabbd14511b9e8a55e92af97fb3a0461-le32d4.cache-3
      1 /home/n/.fontconfig/ce677e824a231659fde4a3a246a10ea3-le32d4.cache-3
      1 /home/n/.fontconfig/d52a8644073d54c13679302ca1180695-le32d4.cache-3
      1 /home/n/.fontconfig/d60319d88cac85ba9e1a07bd06cfbb8c-le32d4.cache-3
      1 /home/n/.fontconfig/ddd4086aec35a5275babba44bb759c3c-le32d4.cache-3
      1 /home/n/.fontconfig/de9486f0b47a4d768a594cb4198cb1c6-le32d4.cache-3
      1 /home/n/.fontconfig/e0853c5e7e7fc9a9e822b52cb2e640cf-le32d4.cache-3
      1 /home/n/.fontconfig/e13b20fdb08344e0e664864cc2ede53d-le32d4.cache-3
      1 /home/n/.fontconfig/e383d7ea5fbe662a33d9b44caf393297-le32d4.cache-3
      1 /home/n/.fontconfig/e7071f4a29fa870f4323321c154eba04-le32d4.cache-3
      1 /home/n/.fontconfig/f24b2111ab8703b4e963115a8cf14259-le32d4.cache-3
      1 /home/n/.fontconfig/f4b919f1221209ef62afab2e83b553ed-le32d4.cache-3
      18 /home/n/.fonts/BurstMyBubble.ttf
      18 /home/n/.fonts/kaileenw.ttf
      18 /home/n/.fonts/King
      18 /home/n/.fonts/MtBdLfRg.ttf
      18 /home/n/.fonts/Paskowy.ttf
      1 /usr/lib32/libfontconfig.so.1.4.4
      18 /usr/share/fonts/opentype/ipaexfont/ipaexg.ttf
      18 /usr/share/fonts/truetype/adf/AccanthisADFStd-Regular.otf
      18 /usr/share/fonts/truetype/adf/BaskervaldADFStd-Regular.otf
      18 /usr/share/fonts/truetype/adf/GilliusADF-Regular.otf
      18 /usr/share/fonts/truetype/adf/IrianisADFStd-Regular.otf
      18 /usr/share/fonts/truetype/adf/LibrisADFStd-Regular.otf
      18 /usr/share/fonts/truetype/adf/MekanusADFStd-Regular.otf
      18 /usr/share/fonts/truetype/adf/SwitzeraADF-Regular.otf
      18 /usr/share/fonts/truetype/adf/Verana-Regular.otf
      18 /usr/share/fonts/truetype/adf/VeranaSansDemi-Regular.otf
      18 /usr/share/fonts/truetype/aoyagi-kouzan-t/AoyagiKouzanFontT.ttf
      54 /usr/share/fonts/truetype/arphic/ukai.ttc
      18 /usr/share/fonts/truetype/freefont/FreeSans.ttf
      18 /usr/share/fonts/truetype/freefont/FreeSerif.ttf
      36 /usr/share/fonts/truetype/kochi/kochi-gothic-subst.ttf
      18 /usr/share/fonts/truetype/komatuna/komatuna-p.ttf
      18 /usr/share/fonts/truetype/linux-libertine/Biolinum_Re-0.4.1RO.ttf
      18 /usr/share/fonts/truetype/linux-libertine/LinLibertine_Re.ttf
      18 /usr/share/fonts/truetype/msttcorefonts/Arial_Black.ttf
      18 /usr/share/fonts/truetype/msttcorefonts/Arial.ttf
      18 /usr/share/fonts/truetype/msttcorefonts/Trebuchet_MS.ttf
      18 /usr/share/fonts/truetype/msttcorefonts/Webdings.ttf
      18 /usr/share/fonts/truetype/openoffice/opens___.ttf
      18 /usr/share/fonts/truetype/takao/TakaoPGothic.ttf
      18 /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf
      18 /usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif.ttf
      18 /usr/share/fonts/type1/gsfonts/b018012l.pfb
      18 /usr/share/fonts/type1/gsfonts/d050000l.pfb
      18 /usr/share/fonts/type1/gsfonts/n019003l.pfb
      18 /usr/share/fonts/type1/mathml/Symbol.pfb
      18 /usr/share/texmf/fonts/type1/public/lm/lmcsc10.pfb
      18 /usr/share/texmf/fonts/type1/public/lm/lmdunh10.pfb
      18 /usr/share/texmf/fonts/type1/public/lm/lmex10.pfb

      Expected behavior:
      Viewer maintains only those font handles it actually needs and uses in the client, so the total count of each should look something like this:
      22 /home/n/bin/SecondLife-i686-2.6.3.0/fonts/DejaVuSans-Bold.ttf
      1 /home/n/bin/SecondLife-i686-2.6.3.0/fonts/DejaVuSansMono.ttf
      1 /home/n/bin/SecondLife-i686-2.6.3.0/fonts/DejaVuSans-Oblique.ttf
      31 /home/n/bin/SecondLife-i686-2.6.3.0/fonts/DejaVuSans.ttf
      1 /usr/lib32/libfontconfig.so.1.4.4

      Steps to reproduce:
      1. Start the viewer and log in
      2. Open a terminal and run `lsof -c do-not` to see all file descriptors used by the viewer, `lsof -c do-not | wc -l` for a total count
      3. Observe that, depending on your available fonts, more than 900 might be in use (try `lsof -c do-not | egrep -o '[^ ]font[^ ]' | sort | uniq -c | less` for a count of each to see how bad it is)
      4. Depending on your systems file descriptor limit per process, after some time APR will begin issuing warnings about too many files open
      5. HTTP requests from the viewer will begin failing, content will stop loading, viewer tries to delete the texture cache repeatedly
      6. Viewer gets logged out or crashes

      The cause of this appears to be the way the Linux client searches for fallback fonts. Searching for fallback fonts was probably necessary in the past because of differences in what fonts were packaged with the viewer, but nowadays all the fonts needed are packaged in and defined in fonts.xml, rendering this processing obsolete – if you really want to change or add fonts, it should be done through some editing of fonts.xml or other .xml files and by including your fonts. It should be noted that on Windows and Mac, there are clear comments stating that the dynamic fallback function is obsolete now that the fonts are defined in fonts.xml; these functions return an empty set.

      The attached patch brought my viewer's average file descriptor count from about 1100 down to 190 during normal use, and it's now far more reliable.

        Attachments

          Activity

            People

            Assignee:
            discrete.dreamscape Discrete Dreamscape
            Reporter:
            discrete.dreamscape Discrete Dreamscape
            Extended Group Visibility:
            jira-users
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                PagerDuty