Lync Server 2010 Troubleshooting Tips

Last updated: 24.10.2012

When deploying Lync Server 2010 you will most likely come across a lot of strange behaviour and stuff that don’t work as expected due to configuration errors or environmental issues. I will in this post try to collect the solutions and articles I find and update along the way, to have troubleshooting tips in one place. Please let me know if any good blog posts or articles that solve some specific problems is not listed here.

General Troubleshooting
Troubleshooting Lync may be a daunting task since there is a lot of elements that is involved in a deployment. Over the years troubleshooting OCS and Lync I have found a routine for eliminating problems. If you have deployed Lync Server and there is connectivity issues or some features not working I usually go about it in the following order: Continue reading

Stale OCS objects showing up in Lync BPA

Recently I ecountered a problem within the Lync Server 2010, Best Practices Analyzer where it found an error for the old OCS 2007 R2 EDGE server.

Problem

  • Lync Server 2010, Best Practices Analyzer returned an error stating the following
    • Error: A trusted service with the type “MRAS” and the fully qualified domain name (FQDN) “” was found for empty or unmatched Globally Routable User Agent URI (GRUU)
  • Searching this up in ADSI edit showed the object present under Trusted Services and aslo under Global Objects container
    • Full path: CN=RTC Service,CN=Services,CN=Configuration,DC=Contoso,DC=com

  • Notice in the picture the yellow highlight that the GRUUID is in capital letters. It turns out that OCS objects are in capital letters and Lync object are in small letters. This is confirmed by Microsoft
  • It is safe to remove these stale objects and OCS objects are easy to identitfy

Solution

  • Delete the stale objects from TrustedServices and Global Objects, containing references to the old EDGE server
  • When running Test-CsTopology again, wich also returned the error code, we got error messages on EDGE server having a missing GRUUID
  • Did rerun Publish-CsTopology
  • The missing objects got recreated
  • Test-CsTopology went through fine
  • Lync Server 2010, Best Practices Analyzer returned a healthy deployment

Conclusion

  • When uninstalling the OCS 2007 R2 services and serves we forgot to remove the OCS 2007 R2 EDGE from the OCS Configuration
    • These settings can be found under Global Properties->EDGE Servers in the OCS GUI
  • The OCS objects should be uninstalled when doing a proper removing of services and features
  • It is safe to remove stale OCS objects that is in capital letters
  • Publish Topology will recreate objects that are missing

NOTE: When you install OCS or OCS R2, the SIP Domain object in the Global Settings container is created as a Version 4 (legacy) object. Be careful not to accidentally delete your SIP Domain object, as it will appear as a legacy object in ADSI edit.

Topology Basics You Must Know Before Planning for Lync Server 2010

In preparation for the Exam 70-665: PRO: Microsoft Lync Server 2010, Administrator I found this article on TechNet useful: http://technet.microsoft.com/en-us/library/gg398552.aspx. It’ s about need to know basics and numbers when planning your Microsoft supported Lync design. Here is a summary with the numbers you must know before taking the Lync PRO Exam:

Site Types

  • Central Site
    • Contains at least one Front End Pool or Standard Edition Server
    • Is mandatory in a Lync Server Deployment
  • Branch Site
    • Is connected to exactly one Central Site and contain one of the following
    • A PSTN gateway and, optionally, a Mediation Server.
      • Designed for branch sites with between 1 and 25 users
    • Survivable Branch Appliance (SBA)
      • Is an industry-standard blade server with a Microsoft Lync Server 2010 Registrar and Mediation Server running on Windows Server 2008 R2
      • The SBA also contains a PSTN gateway
      • Designed for branch sites with between 25 and 1000 users
    • Survivable Branch Server (SBS)
      • Is a server running Windows Server that meets specified hardware requirements, and that has Lync Server 2010 Registrar and Mediation Server software installed on it
      • Does not contain a PSTN gateway
      • Designed for branch sites with between 1000 and 5000 users

Server Roles

  • Standard Edition Server
    • Uses local SQL Express database to host users
    • One Standard Edition server supports as many as 5,000 users
    • In virtual deployments this is reduced to about 2,500 users
    • Can be part of a backup pool topology
  • Front End Server and Back End Server
    • Front End pool is a set of Front End Servers, configured identically, that work together to provide services for a common group of users
      • One Front End pool in the deployment also runs the Central Management Server
      • The Central Management Server also provides Lync Server Management Shell and file transfer capabilities
    • Back End Servers do not run any Lync Server software
      • Can be a single SQL server, but its recommended to run a cluster of two or more
      • Information stored in the Back End Server databases includes
        • presence information
        • users’ Contacts lists
        • conferencing data including persistent data about the state of all current conferences
        • conference scheduling data
    • Front End Server scalability
      • One Front End Server for every 10,000 users homed in the pool
      • In virtual deployments this is reduced to about 5,000 users
      • The maximum number of users in one Front End pool is 80,000, if more than this, deploy an additional pool
  • A/V Conferencing Server
    • It can be collocated with Front End Server, or deployed separately as a single server or A/V Conferencing Server pool
    • One A/V Conferencing Server deployed as a single server for each 20,000 users at a site
  • EDGE Server
    • Server enables your users to communicate and collaborate with users outside the organization’s firewalls
    • Can be deployed with
      • One external IP
      • Three external IP’s
      • Can be behind NAT
      • But when in HA scenario, AV EDGE need to be directly routable
    • Deploy one Edge Server for every 15,000 users you expect to access a site remotely
    • In virtual deployments this is reduced to about 7,500 users
  • Mediation Server
    • Mediation Server translates signaling and, in some configurations, media between your internal Lync Server infrastructure and
      • Public switched telephone network (PSTN) gateway
      • IP-PBX
      • Session Initiation Protocol (SIP) trunk
    • Estimating Voice Usage and Traffic
      • For Light traffic (one PSTN call per user per hour) figure 15 users per port
      • For Medium traffic (2 PSTN calls per user per hour) figure 10 users per port
      • For Heavy traffic (3 or more PSTN per user calls per hour) figure 5 users per port
  • Monitoring Server
    • Monitoring Server collects data about the quality of your network media and collects call error records (CERs), which you can use to troubleshoot failed calls
    • can support up to 250,000 users if not collocated with Archiving Server
    • If collocated, it can support up to 100,000 users
  • Archiving Server
    • Enables you to archive IM communications and meeting content for compliance reasons
    • If you do not have legal compliance concerns, you do not need to deploy Archiving Server
    • Archiving Server can support up to 500,000 users if not collocated with Monitoring Server
    • If collocated, it can support up to 100,000 users
  • Director
    • Can authenticate Lync Server user requests, but do not home user accounts, or provide presence or conferencing services
    • Most useful in deployments that enable external user access, where the Director can authenticate requests before sending them on to internal servers
    • Directors can also improve performance in organizations with multiple Front End pools
    • Deploy one Director for every 15,000 users who will access a site remote
    • In virtual deployments this is reduced to about 7,500 users

Awarded Lync Microsoft Most Valuable Professional (MVP)


Today was a good day! I received an email from Microsoft presenting me with the 2011 Microsoft® Lync MVP Award for the first time. The mail stated: “This award is given to exceptional technical community leaders who actively share their high quality, real world expertise with others. We appreciate your outstanding contributions in Lync technical communities during the past year”. I am proud to be a Lync MVP and want to thank the people within Microsoft who nominated me and the community who share the same passion for technology. It is a good feeling to get recognition for the hard work and dedication you put in to your work and your hobby. I look forward to interact within the Lync MVP community.

View my MVP profile page here: https://mvp.support.microsoft.com/profile/Hansen1

Today there is 42 Lync MVPs and I am one of them. There where 140 new MVP awards and 814 MVPs that were re-awarded world-wide within the whole range of Microsoft technologies. Read more about it here: http://blogs.msdn.com/b/mvpawardprogram/archive/2011/03/28/congratulations-to-the-new-and-re-awarded-mvps.aspx

What is the MVP Award?

The Microsoft MVP Award recognizes exceptional technical community leaders from around the world who voluntarily share their high quality, real world expertise with others. Microsoft MVPs are a highly select group of experts representing technology’s best and brightest who share a deep commitment to community and a willingness to help others. Worldwide, there are over 100 million participants in technical communities; of these participants, there are fewer than 4,000 active Microsoft MVPs.

Read more about it here: http://mvp.support.microsoft.com/

A short video about whats cool about being an MVP

Dedicated Lync Server 2010 Mediation Server with two NIC’s

Background

Recently I implemented a dedicated Lync Mediation server with two NIC’s. In this case I had to use a dedicated Mediation server because the IP Trunk provider was direct SIP. The Mediation server came with one NIC in the server subnet and the other NIC in the telephony subnet.

The problem

After implementing the Lync Topology correct and validated ok, we had one way audio with PSTN calls or no audio at all. All the calls got connected though. I started deep diving into logs and network configuration. While searching the web I found some very good articles about this regarding OCS 2007 R2. I found that there is little or no change in how to configure the Lync Mediation server versus configuring the OCS Mediation server network vise.

The Symptoms

During troubleshooting I saw the following errors:

  • Event ID 25073
    • The Mediation Server service has received a call that does not support comfort noise from the Gateway peer
  • In the SDP packet on the gateway side we saw that RTP traffic was proposed to go over the server subnet IP and not the Gateway subnet IP
    • No audio could be established because of routing issues
  • RTP traffic from the gateway came from an other IP than the SIP IP
    • The Gateway had a media proxy with a different IP than the SIP IP

Reading material

I found these article good reading during the troubleshooting

OCS Mediation Server NIC Configuration: http://blogs.pointbridge.com/Blogs/schertz_jeff/Pages/Post.aspx?_ID=55
Forum discussion about two NIC’s in same subnet: http://social.microsoft.com/Forums/en/communicationsservertelephony/thread/6abb5010-5fd3-4940-8b91-0d2b0184a9fe
Communications Server Mediation Server: Dual NIC Issue: http://technet.microsoft.com/en-us/library/ff518437.aspx

The solution

Make sure the following is true

  • Default gateway is only configured on the NIC in the server subnet, under IPv4 IP configuration
    • Use no default gateway on the Voice Gateway NIC
  • Make sure that the Voice Gateway NIC is on top when running ipconfig
    • If its not, make it so

  • Add static persistent routes for both the SIP and RTP IP against the correct NIC id
    • route ADD 77.88.99.80 MASK 255.255.255.255 10.150.80.241 IF 13 -p
    • route ADD 77.88.99.83 MASK 255.255.255.255 10.150.80.241 IF 13 -p
  • I was also considering the Strong Host model, but that is when several NIC’s need to share a default gateway, don’t think it applies here

Conclusion

In this case it was the missing route to the RTP media proxy IP that was missing. The symptom of the calls being established but no audio is evidence of that SIP traffic is going through ok, but RTP traffic is not being routed correct. The wrong IP being suggested was also evidence towards routing issues. Adding the missing route solved the problem. The Event ID 25073 also disappeared when the routing issue was resolved.

Regarding dedicated Lync Mediation Servers they should have a single NIC when the Voice Gateway is in the same subnet as the Lync Server themselves. Two NIC’s should only be considered when they can be placed in totally different subnets.

Script to enable Global Voice Routing in Lync Server 2010

With Lync Server 2010 more and more deployments involve Enterprise Voice and Pstn access. When implementing in pilots and single site environments my customers tend to want everyone activated being able to dial to the Pstn network regardless of location or position.

With this in mind I have created a script to easily implement appropriate Nomralization Rules, Pstn Usage, Route with Gateway and Voice Policy without any typos. Since all this is intended for global use and the scope is global, all users get access to the Dial Plan and Route. All calls and numbers are routed through a single Gateway.

The script is run at your own risk, tips and feedback is most appreciated

View and download the script here: https://msunified.net/lyncdownloads/script-set-globalvoicerouting-ps1/

Intended Scenario

  • One CentralSite
  • One PstnGateway
  • All Enterprise Voice users will be able to call
  • All previous configuration will be removed

To have a clean as possible configuration some existing default configuration is removed

  • The “Prefix All” Normalization Rule on the Global Dial Plan
  • The Local Route under Routes is removed
  • All the default PstnUsage records are removed
  • Running the script several times does no harm, all error messages are suppressed

The Script creates the following

  • Four Norwegian Nomralization Rules under the Global Dial Plan
    • No need to Grant users with usage
    • Just change and adapt them to your own country/environment before running the script
  • Creates PstnUsage named Global
  • Edits the Global Voice Policy and adds PstnUsage Global
  • Finds the PstnGateway configured in the Topology Builder
    • Only one is added
  • Creates a new Route named GlobalRoute and adds Global as PstnUsage and the PstnGateway found earlier
  • Creates a VoiceTestConfiguration for one of the normalization rules
  • Runs the Test
  • Displays the result

Screenshot

Lync Server Front End: Lost connection to the Web Conferencing Edge Server

I have seen this problem in a couple of deployments now and have decided to share this information since there is not much about it on the web.

The problem

Event ID 41024 – Lost connection to all Web Conferencing Edge Services
Event ID 41024 – Lost connection to the Web Conferencing Edge Server
Event ID 41025 – Connection to the Web Conferencing Edge Server has been restored

These Event ID’s appears every five minutes on the Lync Front End Server, but not in all implementations. I don’t know why it is in only some implementations.

The Resolution

Disable IPv6 on the Front End server. When implementing Exchange 2007 and 2010 we often came across a similar problem where the resolution was to disable IPv6. Disabling IPv6 has cleared the Event ID errors in the cases I have seen. To disable IPv6 properly don’t just uncheck IPv6 on the network adapter, but do it in registry and reboot. See this KB article for how to disable IPv6 properly: http://support.microsoft.com/kb/929852

Screens of the exact Event ID’s

 

 

 

 

 

Script to find available telephone numbers in Lync Server 2010

Having worked on a couple of Enterprise Voice deployments since the launch of Lync Server 2010 I have been missing a feature that I honestly thought would be incorporated by launch. It is the ability to find unused available numbers in a given number range assigned to the Lync Server based on Unassigned Numbers. I even hoped it would be possible to assign a teluri to users through a drop down list in the Silverlight GUI since the information is available in the server. I hope this will be available in a later Service Pack or Cumulative Update.

Since this is a feature I really want, I have tried to create a PowerShell Script to find out how many numbers are available in any given number range and what they are. First of all I want to declare that I am a PowerShell novice so my syntax may not be that optimized. The script works, though there’s a lot of variables and loops that I am sure could be done better. The script does the following:

  • It goes through each entry in Unassigned Phone Numbers that needs to be populated with all numbers series available to the Lync Server
  • Then it gets all teluri’s for all the users, dial-in conferencing and Exchange UM contacts
    • I didn’t find the response group command to get the teluri while I was at it, will add this in an update
  • Then the string gets “washed” for “tel:+47” where 47 is the hardcoded region code for Norway
    • The string also get “washed” for any extension numbers
  • After that a new string gets created to get rid of property types and so on
  • I use Compare-Object to compare the Unassigned Number array with the used numbers
    • Run a check to find what numbers where not used and count them
  • Then I find out what the total number of unused numbers are and write the output

I hope this script will come useful to anyone who needs this feature and that others can use this to and adapt it to their needs. Please share any improvements that could be made to the script. Enjoy!

https://msunified.net/lyncdownloads/script-list-unusednumbers-ps1/

Gartner Magic Quadrant for Unified Communications as a Service 2010

Gartner released their Magic Quadrant for Unified Communications as a Service (UCaaS) 2010 december 2010. This is the second time they release a MQ for this segment. They say it is a market that is in a maturity phase and will continue to change in the coming years. No participants are in the Leaders quadrant this year.

UCaaS is a cloud-based delivery of integrated communications capabilities spanning voice, messaging, conferencing and presence. Businesses should view UCaaS as an emerging alternative to premises-based UC platforms for improving productivity, processes and workflow.

Microsoft is a challenger with its BPOS offering. With Office 365 maybe they can become a leader in the UCasS market as they can offer Lync Online and Office Online in addition to Exchange and Sharepoint. Some of the major improvements in Lync Online is

  • Federation is now possible allowing collaboration with federated contacts
  • Combine Lync Online with On-Premise deployment as you can with Exchange 2010
  • Access to the Lync 2010 client is a benefit in it self

Read the full report here: http://www.gartner.com/technology/media-products/reprints/microsoft/vol14/article13/article13.html