RNUG Lotus User Group (www.vlaad.lv)

    "Panic Invalid pool free chain" after upgrade to 12.0.1 or 12.0.1FP1 technote published

    Vladislav Tatarincev  24 May 2022 09:28:15
    As mentioned in previous post, HCL has published technote about 12.0.1 and 12.0.1
     


    HCL Domino server keeps crashing due to memory corruption with an error "Panic Invalid pool free chain" after upgrade to 12.0.1 or 12.0.1FP1
    https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0098671

    Cause
    Once the debug parameter debug_checkmarkers=1 is enabled on server, the server crashes with an error "PANIC: CheckMarkers: Memory following object has been destroyed" as soon as memory corruption is detected. While reviewing logs, The call stacks in NSD indicate a network buffer has been written past it's boundary.

    Resolution
    There's a change in HCL Domino v12, where we increased the network buffer size from a default of 4000 to 8000 as we noticed some network operations need additional memory. Increasing network buffer further from 8000 to 8100 by making below change in server's notes.ini file, the server stops crashing and gets stable.

    TCPIP=TCP,0,15,0,,12320
    to
    TCPIP=TCP,0,15,8100,,12320

    And adding below parameter too

    debug_net_max_buffer_size=8100

      Network issue crash on DOmino 12.0.1FP1

      Vladislav Tatarincev  20 May 2022 13:51:36
      As you know, in Domino 12, network buffer is increased from 4000 to 8000 bytes. https://help.hcltechsw.com/domino/12.0.0/admin/wn_tcpipportbuffer.html
      Few of my large clients recently performed upgrade from 11.x to 12.0.1FP1 and we got crashes.

      Crashes were bad, because in one case both clustered server were down. To mitigate we changed Transactional Log style/type to Favour Recovery, so DOmino is up faster after crash.
      We opened case in Support and we got recommendation to set optional parameter for network larger than 8000.

      I am publishing call stack, in case you face same issue, you know what to do.  HCL will soon publish technote on this.

      change
      TCPIP=TCP,0,15,0,,12320
      to
      TCPIP=TCP,0,15,8100,,12320
      and also add the line
      debug_net_max_buffer_size=8100



       [ 1] 0x7FFCE7BA5D04 ntdll.ZwWaitForSingleObject+20 (970687300000010,0,0,1A9E3AD50)
       [ 2] 0x7FFCE4A94DAF KERNELBASE.WaitForSingleObjectEx+143 (10,FFFFFFFFFFFFFFF,7FFC00000000,5d50)
      @[ 3] 0x7FFCB903CC0A nnotes.OSRunExternalScript+1546 (5,0,2344,0)
      @[ 4] 0x7FFCB903929C nnotes.FRTerminateWindowsResources+1532 (5,0,0,1)
      @[ 5] 0x7FFCB903AC83 nnotes.OSFaultCleanupExt+1395 (0,3970,0,1A9E3C5B0)
      @[ 6] 0x7FFCB903A707 nnotes.OSFaultCleanup+23 (3970,0,0,7FFCBA0027B8)
      @[ 7] 0x7FFCB90A8FE6 nnotes.OSNTUnhandledExceptionFilter+390 (1A9E3C490,7FFCBA750A68,1A9E3C5B0,FFFFE896E134FC3)
      @[ 8] 0x7FFCB903D92A nnotes.Panic+1082 (1785906EEA0,22588290071B03F,fffe,1785906EEA0)
      @[ 9] 0x7FFCB9043FF3 nnotes.CheckMarkers+323 (1784081D6B8,178427CBA18,2e800003,0)
      @[10] 0x7FFCB906B297 nnotes.OSMemoryLock+135 (d080002,7FFCBBD20D60,1783B9E0018,2e800003)
      @[11] 0x7FFCB9B4490B nnotes.NetGetDataPtr2+27 (2e800003,7FFC2E800003,178427CBA18,2e800003)
      @[12] 0x7FFCBA5D348A nnotes.PortDriver_QueueIO+90 (178427CBA18,2e800003,0,0)
      @[13] 0x7FFCB9B4D56A nnotes.NetQueueIO+1386 (101d484,17800000000,1A9E3003D,178000020DC)
      @[14] 0x7FFCB4BAFBF2 nserverl.DbServer+2002 (1,97980027,1A9E30002,2e800003)
      @[15] 0x7FFCB4BCD0B6 nserverl.WorkThreadTask+1846 (1038,0,1A9E30000,1c4c021e)
      @[16] 0x7FFCB4B81C88 nserverl.Scheduler+888 (0,0,10380034,1783CC57C78)
      @[17] 0x7FFCB904F898 nnotes.ThreadWrapper+264 (0,0,0,0)
       [18] 0x7FFCE79884D4 KERNEL32.BaseThreadInitThunk+20 (0,0,0,0)
       [19] 0x7FFCE7B51791 ntdll.RtlUserThreadStart+33 (0,0,0,0

        Notes on Mac and some specific VPNs. Network_Resiliency=0

        Vladislav Tatarincev  1 April 2022 12:47:14

        While ago during 12.0 we faced issue that after upgrade to V12.0 Notes was able to work with our VPN server.
        Error message pointed to very low TCPIP level.

        and today my colleagues Stanislav Schvachko has found, that issue is finally resolved (actually 8 month ago, but only 70 as of today)
        Just enable Notes.ini parameter Network_Resiliency=0 , that will disable new V12 feature

        Network Resilence feature is v11 functionality, so upon timeout client is not frozen.
        https://help.hcltechsw.com/notes/11.0.0/otherenhancements_notes11.html


        Seems that in V12 this resilence feature was improved, but with some side effects.

        Here is a link to full technote.


        HCL Notes 12 on Mac cannot connect to the server over VPN
        https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0092536

          Updated MAIL 12.0.1 Template to address some issues

          Vladislav Tatarincev  17 March 2022 12:34:33
          One of my customers noticed that when they join meeting, URL or Online meeting is LowerCased. as result also password is lowercased and people fail to join.

          I opened ticked and very fast HCL did changes to template https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0097354
          The following Technotes/SPRs are addressed


          SPR # SVAIC44HYC >>  When a user replies or forwards a message, the 'message replied/forward to' arrow is not displayed in the view/folder column.
          SPR # SAHNCAUBHJ >> When a user creates a new meeting with Webex, the user sees a second meeting entry in the Notes calendar. This happens when the user's Webex user name/mail address is the same as that of Notes.
          SPR # SANECAUH5J >>  Notes rescheduled meetings do not reflect in Webex or Zoom meeting on DOMI.
          SPR # JPKRC5KT3Y >> "Error loading USE or USELSX module: domiListsCollectionsError" when opening new mail12.ntf with older client.
          SPR # ASATC48JKC >> In Notes V12 mail template, hotspot 'Join' beside the location launches the URL link in lowercase. URL parameter, which contains for example the meeting id, are case sensitive, resulting to a meeting that could not be joined.

            Reading MTDATA.nsf (Message Tracking) and generating your own reports

            Vladislav Tatarincev  13 October 2020 08:22:10
            I did Domino Administration training last week and did it online/remotelly which was new experience for me.
            Course name is Domino ADVANCED training, so students learn real life stuff, and not only basic, but also Advanced things like Verse on Prem, Clusters, Traveler, V10/v11 Upgrade, Performance Tuning.
            Www.cyone.eu/Training explains it in more details.

            When we worked with Message Tracking , and one students asked if he can create custom reports. I do remember that IBM Technote existed that described how to access mtdata.nsf with LotusScript.
            I was unable to find this technote, since IBM removed this from knowledge base. I opened case with HCL and they pointed me to right article in Domino Wiki.

            https://ds-infolib.hcltechsw.com/ldd/dominowiki.nsf/dx/Reading_the_MTSTORE_Database

            To avoid performance issues , I do suggest to archive MTDATA.nsf, and in script depending on search range access only one database, or/and archive.
            Hope it helps.


            Join https://en.RNUG.ru online conference for more tips , latest roadmaps, workshops.

            vlad

              Russian Notes User Group online conference (content in English and Russian) 9-16 October 2020

              Vladislav Tatarincev  22 September 2020 13:19:26
              Russian Notes user conference will be online event, sessions will be in English and Russian
              We will have fantastic sessions from HCL, panagenda, YTRIA, TeamWorkR, CYONE and other business partners.

              Event participation is free, if you want to be a speaker call for abstracts is open until Friday evening, sponsors - great opportunity to present your solution or service.

              Https://en.RNUG.RU/  From Russia with Love.


              Avoid deleting users who left company from Reader/Authors fields and NAMES fields

              Vladislav Tatarincev  15 September 2020 14:11:22

              If you have ACL ACTION "MODIFY ALL NAMES FIELDS, OR READERS AUTHORS FIELDS"  then upon user rename, user is renamed by adminp in names fields and security fields.

              Drawback is when user is leaving company and admin deletes user, if you have this setting enabled, then user will be deleted from all Names fields or Readers/Authors fields.

              Which may lead to strange documents, like Approved by "Empty string", because person who approved left company and adminp deleted him from field.
              Which is not ok, since breaks logic and compliance, we have to know who accepted document.

              I got request from customer to modify my TDI script, to avoid this, and I did some reasearch on this.


              I did some reverse engineering of Domino server code and found, that there are two undocumented parameters that made me very curious.

              ADMINP_DISABLE_NAMEITEM_DELETE
              ADMINP_DISABLE_READAUTH_DELETE


              I opened ticket :) in HCL Support and HCL Support was fast like a rocket!  I got reply from Somnath and Development team that these two parameters will extend functionality of Delete action.

              If you want rename, but want to skip DELETE in Names fields or Reader/Authors fields just enable these parameters. ADMINP_DISABLE_NAMEITEM_DELETE=1
              ADMINP_DISABLE_READAUTH_DELETE=1



              HCL Will publish a technote on this topic today.   This is brand new, so before putting in production do proper tests in test environment!

              Vlad

              I will be speaking on Russian Notes user group event, so everybody is welcome, content in English and Russian  https://en.rnug.ru/

                v11 HTTP Rule bug and workaround

                Vladislav Tatarincev  25 February 2020 09:39:52
                One way to improve HTTP application is to cache different objects, like CSS, JPG , .JS files.
                To do so you must create a HTTP Header rule.



                Image:v11 HTTP Rule bug and workaround
                I always do this when improve performance of HTTP, but in v11 I tried and I got error message, that "Expiration days is required"

                Image:v11 HTTP Rule bug and workaround
                I opened case with HCL support and they confirmed issue, and next day I got info that it will be fixed in 11.0.1 under  SPR # SPPPBLXKG9
                Meanwhile workaround is to do the following


                1. Open names.nsf\pubnames.ntf in Designer.
                2. In Forms, select and edit the form "(Web Rule)"
                3. In the Basic tab of the "(Web Rule)" form, change the "Input Translation" of below two fields.
                  a) Hdr_Ex_Days => Remove the current formula and update it with the below one.
                @If((MappingType = "4" & Hdr_Ex_En != "0" & Hdr_Ex_Opt = "0") | [email protected]; @ThisValue; 5)
                  b) Hdr_Ex_Date = > Remove the current formula and update it with the below one.
                @If((MappingType = "4" & Hdr_Ex_En != "0" & Hdr_Ex_Opt = "1") | [email protected]; @ThisValue; @Tomorrow)
                4. Once done, save & close the form.

                Archives