1.0 About Persistent Search
Persistent Searches, not to be confused with Persistent Connections. Persistent Searches are defined by IETF Draft as part of the LDAP RFC 3377 maintained by the IETF. The definition of, and rules applying to, persistent searches in LDAP are managed by this RFC and not by product vendors. See the Intended Uses within the RFC for a complete description.
As a caution, the RFC memoranda stated the following in the Implementation Considerations:
Each active Persistent Search request requires that an open TCP connection be maintained between an LDAP client and an LDAP server that might not otherwise be kept open. Therefore, client implementors are encouraged to avoid using Persistent Search for non-essential tasks and to close idle LDAP connections as soon as practical.
Novell Corporation offers a competitive “eDirectory” Directory Server. They offered the following guidelines on persistent searches
Persistent Searches are handled by the nsslapd-maxpsearch property in Sun Java DS: Defines the maximum number of persistent searches that can be performed on the Directory Server. The persistent search mechanism provides an active channel through which entries that change (and information about the changes that occur) can be communicated. Because each persistent search operation uses one thread, limiting the number of simultaneous persistent searches prevents certain kinds of denial of service attacks. [mike_webb]
Property Value Entry DN: cn=config
Valid Range: 1 to maximum threadnumber
Default Value: 30
Syntax: Integer
Example: nsslapd-maxpsearch: 30
It is quite apparent that a client implementation that generates a large number of persistent connections to a single Directory Server may indicate that the LDAP protocol may have not been the correct transport. However, horizontal scaling using Directory Proxy Servers, or a LDAP Consumer tier, to spread the load may assist.
The best solution, from an LDAP implementation, would be to limit persistent searches.
2.0 Persistent Searches and Access Manager
UPDATE: (09/09/06)
Starting from Access Manager 7.0 patch4 or above , these persistent search connections can be selectively disabled by setting the following property in the Access Manager’s AMConfig.properties file
To disable all the persistent search connections set the following
com.sun.am.event.connection.disable.list=aci,sm,um
To disable Persistent Search 1
com.sun.am.event.connection.disable.list=aci
To disable Persistent Search 2
com.sun.am.event.connection.disable.list=sm
To disable Persistent Search 3
com.sun.am.event.connection.disable.list=um
Access Manager/SDK(for eg: pre 2.2 J2EE agents) uses persistent searches to notify the changes in the Directory server back to the Access Manager/SDK. A vanilla Access Manager creates three psearch connections during server startup.
2.1 Persistent Connections by Access Manager Server
Persistent Search 1: To notify any changes in the attribute “aci”
This thread is submitted with LDAP filter (aci=\*), it listens for changes in aci and relays the changes back to Access Manager inorder to refresh the cache. The nsIdletimeout is not applicable for psearch connections. This connection will be closed when the DS/AM goes down or in the event of LB/Firewall TCP timeout.
[12/Dec/2005:10:56:45 -0800] conn=223 op=-1 msgId=-1 - fd=25 slot=25 LDAP connection from 10.18.10.190 to 10.18.10.190
[12/Dec/2005:10:56:45 -0800] conn=223 op=0 msgId=4 - BIND dn="cn=dsameuser,ou=DSAME Users,dc=red,dc=iplanet,dc=com" method=128 version=3
[12/Dec/2005:10:56:45 -0800] conn=223 op=0 msgId=4 - RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=dsameuser,ou=dsame users,dc=red,dc=iplanet,dc=com"
[12/Dec/2005:10:56:45 -0800] conn=223 op=1 msgId=5 - SRCH base="dc=red,dc=iplanet,dc=com" scope=2 filter="(aci=\*)" attrs="objectClass" options=persistent
Persistent Search 2: To notify the changes in the service nodes.
In the AM all the service nodes are stored in the directory as an object with marker objectclass sunService or sunServiceComponent. This psearch thread notify the AM about the changes that happens only on the nodes which has objectclass sunService or sunServiceComponent. Any other changes in the directory will be ignored by this psearch.
[12/Dec/2005:10:56:45 -0800] conn=225 op=-1 msgId=-1 - fd=27 slot=27 LDAP connection from 10.18.10.190 to 10.18.10.190
[12/Dec/2005:10:56:45 -0800] conn=225 op=0 msgId=8 - BIND dn="cn=dsameuser,ou=DSAME Users,dc=red,dc=iplanet,dc=com" method=128 version=3
[12/Dec/2005:10:56:45 -0800] conn=225 op=0 msgId=8 - RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=dsameuser,ou=dsame users,dc=red,dc=iplanet,dc=com"
[12/Dec/2005:10:56:45 -0800] conn=225 op=1 msgId=9 - SRCH base="dc=red,dc=iplanet,dc=com" scope=2 filter="(|(objectClass=sunService)(objectClass=sunServiceComponent))" attrs="objectClass" options=persistent
Persistent Search 3: This search is responsible to notify the changes on the entries that does not meet the filter criteria of psearch 1 and psearch 2.
For instance if the users’ telephonenumber is changed, the psearch 1 and psearch 2 does not notify the AM about this change, because the ldap search filter does not match user entries. Hence this particular psearch is required.
[12/Dec/2005:10:56:45 -0800] conn=224 op=-1 msgId=-1 - fd=26 slot=26 LDAP connection from 10.18.10.190 to 10.18.10.190
[12/Dec/2005:10:56:45 -0800] conn=224 op=0 msgId=6 - BIND dn="cn=dsameuser,ou=DSAME Users,dc=red,dc=iplanet,dc=com" method=128 version=3
[12/Dec/2005:10:56:45 -0800] conn=224 op=0 msgId=6 - RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=dsameuser,ou=dsame users,dc=red,dc=iplanet,dc=com"
[12/Dec/2005:10:56:45 -0800] conn=224 op=1 msgId=7 - SRCH base="dc=red,dc=iplanet,dc=com" scope=2 filter="(&(objectClass=\*)(!(|(objectClass=sunService)(objectClass=sunServiceComponent)(aci=\*))))" attrs="objectClass" options=persistent
2.2 Persistent Search connection by LDAPv3 Plugin
If you have created a datastore against a LDAP server which supports RFC2026 then one more psearch connection will be created with basedn configured in the LDAPv3 configuration. For this connection filter would be (objectclass=\*).
Note: This is a very genereous filter, customer may not like their AM servers flooded with many notifications which are not relevant, this would cause unnecessary overhead for the plugin to process unwanted psearch notifications. AM should provide a configurable search filter for this psearch[in AM 7.1 this is configurable]
2.3 Persistent Search connection by standalone Java clients
If the amadmin CLI is invoked this would initiate 3 psearch connections to the directory server these are similar to the one that initiated by the vanilla server.
2.4 Persistent Search connections by pre 2.2 J2EE agents
J2EE agents version 2.1 creates three persistent search connections with same search filters described in the vanilla server category. Since Access Manager and its clients depends on these psearch connections, it is imperative that the underlying data repository server has to be tuned accordingly. There are occasions that the DS could run in to saturations in supporting the psearch connections. In such scenario the AM and its clients has to orderly bailout instead of running in to a deadlock loop. To avoid this ugly scenario besides tuning the directory server you could also tune the Access Manager appropriately. If the deployment uses firewall or loadbalancer in front of DS then the following tuning configuration is a must to avoid the cache inconsistency.
2.4 Persistent search specific properties
2.4.1 Properties that goes in AMConfig.properties
com.sun.am.event.connection.idle.timeout=(Value in minutes)
If the persistent search connections are made through a loadbalaner or firewall then these connections are subject to the tcp time out value of the respective LB and/or firewall. In such scenario once the firewall closes the psearch connection due to a idle tcp time out, then the change notifications cannot happen to AM unless the psearch connection is re established. Customers could avoid this scenario by configuring the idle timeout for the psearch connection so that it would restart psearch TCP connection before the LB/Firewall idletimeout, that way firewall/LB will not have a idle psearch connection.
‘com.sun.am.event.connection.idle.timeout’ specifies timeout value in minutes after which the persistent searches will be restarted. Ideally, this value should be lower than the Load Balancer/ Firewall TCP timeout, to make sure that the persistent searches are restarted before the connections are dropped. A value of ‘0’ indicates that these searches will not be restarted. By default the value is ‘0’
Note: Only the connections that are timed out will be reset. You should never set this value to a low value than the firewall/LB timeout. The delta should not be more than 5 minutes. If your LB’s idle connection time out is 50 minutes then set this property value to 45 minutes.
com.iplanet.am.event.connection.num.retries=3
com.iplanet.am.event.connection.delay.between.retries=3000
com.iplanet.am.event.connection.ldap.error.codes.retries=80,81,91
The above three properties can be used to restart the event service connections in an orderly manner when the DS is down or running out of resources. By default attempt to create event service(psearch) connections will be made for every three seconds until it gets a successful connection based on the error codes. . The time delay between the retry would be 3000 milli secs. You can put any valid LDAP error code(for e.g.: 51) in com.iplanet.am.event.connection.ldap.error.codes.retries property.
All the above three properties are pertaining to event service only these are not shared by any other modules. For example setting idletime out will not affect the SDK ldap connection pool or the AuthN/Policy LDAP connections.
2.4.2 LDAPv3 Plugin specific property
sun-idrepo-ldapv3-config-idletimeout(Persistent Search Maximum Idle Time Before Restart)
It is used by the LDAPv3 datastore to reset the psearch connection made by the datastore instance. Again if you are using a LB or Firewall make sure this value is lower than the LB/Firewalls’ connection idle timeout. It is worth to remember that each instance of LDAPv3 plugin creates a psearch connection. Exercise caution in creating LDAPv3 datastores. Note: Directory Server does not return an error if the base DN of the persistent search does not exists so make sure you supply the correct base DN.
2.5 Trouble Shooting
If you notice the following message in amEventService debug log
netscape.ldap.LDAPException: Error result (51); too many simultaneous persistent searches.
it is quite likely your event service trying to create too many psearch connections or DS already reached the max limit on the psearch connection.
Solution: Verify the AM/SDK configuration make sure no misconfiguration. If the problem persists, look at the DS logs to see how many psearch connections are currently being used. If required tune the max-psearch limit.
Investigate the event service connections and make sure they are up and receiving notifications. In a multiserver deployment scenario, if you have updated something from one server, but this change is not seen in the other AM server. The change notification may not have been received by the AM server where the changes are not seen. To debug this issue first thing you would investigate is to make sure the persistent search connections are established properly Often times a brief network issue could cause these connections to terminate(even though there is a retry mechanism, it betteryou check it out when you sense a brief network outage) . If your Directory Servers are behind the LB/FireWall then make sure the property described in section 2.4.1 configured properly.
3.0 References
Indira's Blog 
TEST
A new property "com.sun.am.event.connection.disable.list" became available on 6.1patch18, 7.0patch2 and 7.1RTM. There are three valid values – aci, sm and um (case insensitive). Multiple values should be separated with ",". This is used to specifies which event connection to be disabled. If no direct modification on the DS normally, you could disable all three persistent connections.
Thanks for the useful info.
One thing though – persistent search 1 (‘aci’) actually reports changes to any entry with an ‘aci’ attribute, even changes that don’t affect ACIs.
Also, it’d be REALLY nice if the cache could be automatically disabled while any persistent searches are down.
Correction: It should be "Higher" instead of existing "Low"
Note: Only the connections that are timed out will be reset. You should never set this value to a
<<<< Low>>> value than the firewall/LB timeout. The delta should not be more than 5 minutes. If your LB’s idle connection time out is 50 minutes then set this property value to 45 minutes.
cheers,
Sudhir Kulkarni