Improper/wrong Port-mirroring/SPAN network voice traffic
Protocol not supported
Interface chosen in Sniffer configuration page
- Solution : While using Multi-Interface machine, please make sure the interface chosen in the VQManager Sniffer Configuration wizard receives all the network Voice traffic. Select the right interface that receives all the VoIP traffic (refer to "Configuring Sniffer" in the Help document).
Make sure a VoIP Call is started in the network after configuring VQManager. Only after updating the Sniffer Configuration will VQManager show call data for the calls. Refresh the page to check for any call information to be displayed
Ports in which Signaling traffic transferred
- By default, VQManager listens for SIP @ 5060 port, SCCP @ 2000 & H.225 @ 1720. These are the default values suggested in the Protocol spec.
- Solution: If you are using a different port please specify the same in the 'Admin Tab' -> Sniffer -> 'Protocol Settings'. If you are not aware of the port, please enable 'All Port' in the same page.

NOTE: Enabling 'All Ports' in high traffic environment will add additional load on VQManager call processing and will impact performance of VQManager

Calls imported by CDR logs
- Solution : While using the CDR import option for call processing, please make sure that the imported CDR data is current date. Since VQManager Monitor tab opens up with the current day, if the reports has been imported for previous dates, change the calender to view the appropriate date. Refer our help document for Calender http://www.manageengine.com/products/vqmanager/help/working/setting_client_view_port.html#__setting_calendar
- Currently, VQManager supports the CDRs generated by Cisco Call Tracker, Cisco Unified Communications Manager 6.1, Cisco Call Manager 4.0 & 5.0, Shoretel 6.1, Vovida Vocal, Asterisk, Avaya S8700, and much more formats. This link provide more details about CDR formats supported by VQManager: http://www.manageengine.com/products/vqmanager/help/Configuration_wizard.html#cdr-import . If you are using any different formats of CDRs in your network, please send us the sample CDR files and any document explaining CDR fields to our VQManager support. We will analyze those formats in our local labs and provide our inference on supporting those in VQManager.

1b. Few calls in the network are not reported by VQManager?

Port Mirroring
Hybrid Signaling Protocol used
Ports in which Signaling traffic transferred
- By default VQManager listens for SIP @ 5060 port, SCCP @ 2000 & H.225 @ 1720. These are the default values suggested in the Protocol spec.
- Solution: If you are using a different port please specify the same in the 'Admin Tab' -> Sniffer -> 'Protocol Settings'. If you are not aware of the port, please enable 'All Port' in the same page.
- NOTE: Enabling All Ports in high traffic environment such as more than 200+ concurrent calls may impact performance of VQManager call processing
Check Licensing options

1c. What is Unmonitored Calls in VQManager?

Calls will be moved to Unmonitored state in case of complete call signaling packets(Call initiation and termination traffic) not reaching VQManager sniffer listening interface (NIC). It may be due to improper port-mirroring/SPAN or installing VQManager in a wrong place.
VQManager, by default checks if there exists an interruption of Call signaling traffic for 60 seconds and end the call stating Unmonitor due to no media transmission with Call status 8003. User can increase this default check from 60 seconds or even disable this configuration by configuring 0 seconds in Sniffer configuration advanced settings page.
This link provides the list of Custom Codes used in VQManager.

Also refer What is Unmonitored due to no media transmission(8003)?, What is 'Media Transmission stopped' calls(8100)?

The codes 8003 and 8100 used in VQManager are not SIP/SKINNY/H.323 codes. These codes are used by VQManager in some special cases to mark a call as either 'Completed' or 'Unmonitored' based on the standard call flow. Please refer the above links for more information on when these codes are used.

1d. What is Unmonitored due to no media transmission(8003)?

If no signaling packets are received by VQManager for a certain time interval, then the call would be moved to the 'Unmonitored calls' category.

Possible reason: Could be due to improper port mirroring

Workaround: Try configuring "Unmonitor calls with no signaling packets for" value to '0'(zero) seconds in the 'Admin Tab' -> Sniffer -> 'Advanced Settings'. This will stop VQManager marking calls as Unmonitored.

NOTE: By setting 'zero' value in the Sniffer settings may lead few calls to be in Active state if the respective call tear down packets did not received in VQManager NIC.

1e. What is 'Media Transmission stopped' calls(8100)?

Cause : These type of calls were reported by VQManager based on Call Media(RTP/RTCP) interruption. This will occur in case of either initiator/ Participant traffic which is not port-mirrored to VQManager, or due to abrupt termination of call due to power loss and so on. By default, VQManager checks if there exists media transmission between Initiator and Participant of a call. If the media transmission is interrupted for more than 30 seconds without receiving any call termination traffic, then VQManager treats the call as terminated and ends the call stating Media Transmission stopped with Call Status 8100.

Workaround : User can change this default setting by increasing the default value of 30 seconds or by disabling this setting by configuring the value as 0 seconds in "End calls whose RTP flow stopped for" value under 'Admin Tab' -> Sniffer -> 'Advanced Settings'.

NOTE: Setting 'zero' value in the Sniffer settings may lead few calls to be in Active state if the respective call tear down packets are not received in VQManager NIC.

1f. Active calls do not get terminated in VQManager/ Long duration active calls

Probable reason for Long duration active calls shown in VQManager is, due to some network traffic issue, VQManager listening interface does not received the Call tear-down packets such SIP-BYE in case of SIP.

In few networks there may exist erroneous/spurious signaling packets. Processing these packets will result in error and the active calls present at that instance may not end properly. Please send the VQManager support log files from 'Support' tab -> by clicking 'Create' latest support file to our support.

Try the workaround of changing the "Unmonitor calls with no signaling packets for" under 'Admin Tab' -> Sniffer -> 'Advanced Settings' to Zero['0'] . If the issue still persists, please contact VQManager support with packet capture of few completed calls being shown in active calls in VQManager.

NOTE: Changing '0'(zero) seconds value in the Sniffer settings may lead few calls to be in Active state, if the respective call tear down packets is not received in VQManager NIC.

1g. Calls get split into two/many in VQManager

Hybrid Environment - Please check whether you have a Hybrid environment - Using more than one Signaling protocols. In the above case, there is a possibility of Calls getting split, please contact VQManager support team with appropriate call captures.

Duplicate packets due to Port Mirroring - While spanning the Voice traffic to VQManager, please make sure that there is no chance for packet duplication. Check whether you have mirrored Ingress/Egress/Both traffic to VQManager listening NIC.

In few VoIP environment such as Asterisk Server, the call server itself will participate in the Signaling and Media traffic. And hence intentionally split the call into 'A -> Asterisk' and 'Asterisk ->B' by having separate call-identifier for each call leg.

1h. How to see quality for each half of the call?

To see the Voice Quality statistics for each half of each call, click on the particular call and then click on the "View Trend" links available under the names of initiator and participant of the call. The QoS details will be opened in a new window. Refer the below screenshot for more information.

If you click on the view trend of the initiator of a call, it will show the Quality of the call as heard by the initiator and viceversa.
For example, if the QoS for initiator view trend of a call is poor then it means that the initiator is not able to hear the voice of the participant properly and there is some problem in the voice transmission from the participant of the call as it reaches the initiator.

2. QoS

2a. Qos is not reported for all calls while using Sniffing

QoS not reported for any call

VQManager fetches QoS information(MOS, Delay, Jitter, Loss, etc.,) from the media packets [RTP /RTCP] traversed between the IP phones. Please verify and ensure your port mirroring setup by which you have routed the Voice[RTP/RTCP] traffic to VQManager listening NIC. Check the presence of RTP and/or RTCP packets in the Call Flow diagram of Call details page.
Bidirectional Media Traffic

2b. Delay, Jitter values reported by VQManager does not match with user experience

Based on VQManager QoS calculation:

Possible Cause I - Erroneous RTCP packets: By default VQManager calculates the QoS from the RTCP packets, as the RTCP packets are generated by the Endpoints, these will be more accurate and will reflect the user experience closely. In few cases there may be a chance of miscalculation in the RTCP calculation logic itself. Please check with your VoIP provider regarding incorrect RTCP packet generation by the IP phones.
Possible Cause II - In the absence of RTCP, VQManager will try calculating the best possible QoS values from the RTP packets from the VQManager Sniffing location. Again this may differ from actual information as VQManager will be sniffing the RTP packets either from Initiator or Participant end.

It is not recommended to calculate Delay, Jitter values using 'ping' request to the IP address and compare those values with VQManager reported QoS values, the results will not be same. As the Delay values reported by VQManager is the delay taking place when the call is Live. When you 'ping' the particular device for delay, it may have different value at that time and hence cannot be compared with the delay reported when the call was live. Also there are some other factors like short-path routing and setting priority levels for VoIP and other data packets could result in mis-match of ping-delay and VQManager-reported-delay. Also check Huge jitter values reported in VQManager

Please let us know 'how did you calculated' Delay, Jitter values and trying to compare those values with VQManager for us to provide more details.

2c. Only one call leg contains QoS values, why?

Possible reason for this could be the one way RTCP information. As VQManager receives single side RTCP, it will report the QoS for Initiator or Participant of the call from where it received the RTCP. Please check whether the Call Flow diagram shows bidirectional RTCP packet, if you noticed bidirectional RTCP, contact VQManager support with appropriate packet capture[How to get packet capture]

2d. Few calls do not have QoS Values / Long duration completed calls doesn't have QoS

Duration of the call: It should be at-least greater than 30 seconds. For short duration calls the RTCP packet could not have been generated.

Single directional media traffic - Even for longer duration calls QoS will not be plotted. May be due to improper port mirroring, single side RTP alone would have been mirrored to VQManager listening NIC. Its mandatory to have bidirectional RTCP and/or RTP for VQManager to calculate the QoS parameters. If you have mirrored the RTP and RTCP, please check for the presence of bidirectional RTP and/or RTCP packets in Call Flow diagram of Call details page. VQManager will report on QoS for calls having bidirectional RTP and/or RTCP packet flow. Refer "How VQManager Calculates QoS " for more information on precedence of RTCP over RTP.

2e. How is QoS calculated in VQManager?

By default, VQManager calculates the QoS(Delay, Jitter & Loss) parameters from the RTCP packets. As the RTCP packets are generated by the Endpoints, these will be more accurate and will reflect the user experience closely. In the absence of RTCP, VQManager will calculate the best possible QoS values by parsing the RTP(data) packets transferred in between the IP Phones. MOS scores are derived based on the Simplified E-Model (defined by ITU - http://www.itu.int/ITU-T/studygroups/com12/emodelv1/index.htm) that takes into consideration network quality parameters such as Delay, Codec used, Packet Loss and Jitter. For more detailed explanation and to check the formula used for MOS calculation please refer Advanced QoS calculation logic in VQManager .

2f. Call shows high percentage of packet loss

Possible reason for HIGH packet loss information shown in VQManager could be due to the packet duplication in VQManager listening interface. One can verify this by checking the duplicated entries for Signaling packets in the Call Flow diagram of Call Details page. But in some cases, media(RTP/RTCP) packets alone may get duplicated, which cannot be viewed using Call Flow diagram. From VQManager 6.2 release onwards, a fix for discarding the duplicated media stream has been added. This could avoid such incorrect loss values reported. If user faces incorrect packet loss details in builds 6.2 & above (build number 6210 & above), Please contact VQManager Support with the completed calls packet capture.

2g. Huge jitter values reported in VQManager

If you are noticing Jitter values in the order of few hundred thousands, please check for the presence of RTCP packets in the Call Flow diagram for that call. As inferred from few of our customer environment, it seems there is a remote possibility of IP Phones(Endpoints) generating/sending spurious/incorrect Jitter values. For cross checking the same, disable the RTCP based QoS Calculation logic in Sniffer Configuration wizard and check with Jitter values for newly made calls. Also, check with the IP Phone vendor regarding the RTCP Jitter calculation, since few software upgrade in the IP Phones may get ride of this issue.

3. Sniffing

3a. Unable to find interface details while configuring Sniffer

Unable to update Sniffer Settings / Protocol detection in Sniffer settings does not work for 10 seconds

This problem is caused due to incompatible winpcap library file. The version of winpcap installed in your machine could be beta version or older version. At present, VQManager does not support beta version of WinPcap libraries. VQManager uses WinPcap libraries for fetching the data from the interface card. It is having some issue in reading the information from NIC and in-turn stops the call processing (data collection) thread of VQManager.

Solution: Install standard Winpcap from http://www.winpcap.org/archive/4.1.1-WinPcap.exe by un-installing other existing winpcap libraries from the machine for proper call processing with VQManager. Also, re-boot the machine to make the changes more effective.

3b. Interface chosen shown as 'null'

If you notice "null" values for the interface already configured in the the Sniffer configuration wizard, please try the below mentioned work-around

Work-around :

Probable cause for this could be your session in the web browser could have been timed-out. Please Sign-out and re-login into VQManager web-client(http://localhost:8647) and try performing the operation.
Interface card could have been unplugged intermediately during the sniffing is running. This will make the VQManager loose the hook between the NIC card and the sniffer. As a work-around, try restarting(stop and start) the VQManager server, this could establish a new hook for sniffing.

3c. No data in traffic monitor graph in the 'Monitor Tab'

The possible cause for the issue is, VQManager sniffer listening interface(NIC) may got interrupted[down].

Workaround : Try reconfiguring the 'Sniffer' settings under 'Admin' tab in VQManager web-client. And if you are still unable to monitor the traffic in VQManager, try re-starting VQManager server. If you are still facing the same issue, check the protocols supports section for supported Signaling and Registration protocols.

3d. How do I check if all the port-mirrored traffic approach VQManager listening interface?

If you are configuring the Sniffer for the first time, VQManager will automatically detect the traffic coming to all the interfaces listed and and shows the corresponding protocols with tick marks as shown below. If you are reconfiguring the sniffer, then click on the "Detect" button available at the top right corner as shown in the below screenshot. If tick marks are not shown, it means there is no traffic coming to the interfaces listed.

4. Configurations

4a. Mails are not delivered in reports and alarms

VQManager sends various notifications to the users (for example, reports & alarms) using an SMTP mail server running in user network.

Solution: For the proper functioning of the E-Mails alerts, you need to configure the SMTP mail server settings in the VQManager. Please refer our help document on configuring Mail Server Settings http://www.manageengine.com/products/vqmanager/help/working/admin_operations.html#__Mail_settings

4b. Support log files are not getting uploaded

While reporting issues or any functionality break to the VQManager team, the user has to upload the logs created by the VQManager application for analysis.

Solution: For uploading logs using 'Create' button in the Support Tab, VQManager needs the Proxy-server for FTP[ Internet access is needed]. In your enterprise network setup, you might need to go through a proxy server to access the internet. In such a case, you need to configure the username and password for internet access. Please refer our help document on configuring Mail Server Settings http://www.manageengine.com/products/vqmanager/help/working/admin_operations.html#__Proxy_settings

4c. No traps have been sent for generated alarms

The administrator can enable the VQManager to generated SNMP traps on specific alarms. This will help in integrating the VQManager alarms with any higher-level NMS like ManageEngine OPManager. The VQManager allows the administrator to set up the list of SNMP trap listeners to forward the SNMP traps.

Solution: By specifying the required SNMP parameters (viz., hostname, port, community, version etc.,), the embedded VQManager SNMP agent can generate appropriate SNMP traps and forward to the defined entries. http://www.manageengine.com/products/vqmanager/help/working/admin_operations.html#SNMP_trapforwarding

4d. Unable to view history data older than a week

Solution : By default, VQManager cleans up data older than 1 week. The clean-up is done by default everyday at 1:00 am. After clean-up, the database would only have data for the last one week from clean-up date and time. Please use the below help link for more information on Database-Clean-Up http://www.manageengine.com/products/vqmanager/help/working/admin_operations.html#__database_administration

Info : In a typical production environment, VQManager deals with a huge amount of data related to calls. The data gets piled up in the database and over a period of time, it becomes too high and might have a bearing on the performance of VQManager. Periodic database clean-up will help maintain VQManager performance. It is recommended to clean-up data older than a week, unless there is strong requirement to keep the data.

4e. Unable to view any SIP registrations under the Endpoints tab registration request

Have you enabled "Stored Registrations" while configuring Sniffer configuration wizard?. If so, please enable it through reconfiguring the Sniffer wizard and observe the behavior. By default, Stored Registration is disabled.

If you are still facing the same issue, check the protocols supports section for supported Signaling and Registration protocols.

4f. Moving the VQManager database to another hard drive/partition location in the machine

Procedure : Please follow the below procedure to store the VQManager database in your preferred location rather than the default one.

Stop the VQManager server if it is running.
Copy and paste the complete /mysql/ directory to your new hard drive location.
Navigate to /bin directory.
Take the backup of startDB.bat file.
- Edit the startDB.bat file by changing --datadir = >mysqldata in the last line
- like @start "MySQL" /B "Û_HOME%inmysqld-nt" --defaults-file=%MYSQLCONFFILE% --low_priority_updates=1 --standalone --basedir="Û_HOME%" --tmpdir="Û_HOME%TEMP" --port="Û_PORT%" --innodb_file_per_table --datadir=">mysqldata"

Save the changes and Try starting the VQManager server.

4g. How do I take backup and restore the VQManager Database(DB)?

Procedure to take a backup of the Database:

Stop the VQManager if it is running.
Open command prompt and navigate to <VQManager_Home>/bin directory.
Execute the command backupDB.bat in windows or backupDB.sh in linux platforms
This creates the backup of the DB in a zip file(with the format yymmdd-hhmm.zip) under <VQManager_Home>/Backup directory (this process will take few minutes depending on the size of DB).
Start the VQManager

Procedure to restore the Database backup:

Stop the VQManager if it is running
Open command prompt and navigate to <VQManager_Home>/bin directory.
Execute the command restoreDB.bat in windows or restoreDB.sh in linux environments, with the back up file name as argument (this process will take few minutes depending on the size of DB).
- C:ManageEngineVQManagerbin>restoreDB.bat ..Backup100401-1530.zip
NOTE -1 : On restoring, the data collected in between the backup time and current time will be dropped. VQManager will have data restored from backup file alone once after restoration.
NOTE -2 : Restoring data across different build number may cause incompatibility issues. Please contact VQManager support team mentioning your requirement in detail.

4h. How do I increase the Session Time out duration?

While accessing VQManager using webclient(http://localhost:8647), the session time out value for every authentication by default is set as 30 minutes.

Procedure : Please follow the below steps for modifying the same.

Open the web.xml file located in /conf directory.
Search for the line '30' in the file and edit the value in the code : 30 . The value is represented in minutes. (You can enter a value up to 1440 minutes. This will set the session timeout value to one day.)
Save the file.
Restart the VQManager server. After this, the session will be maintained for the number of minutes that you have set.

4i. How do I import PhoneIds/Endpoints in the EndPoints Tab?

The phone IDs automatically discovered by the VQManager will be those of unique endpoints (hard phones, soft phones, call servers, media proxy servers, gateways and other VoIP devices) participating in calls in the network collected via Sniffing interface.

Procedure to Import Endpoints

Info: If you import the specific set of Endpoints available in your environment (through CSV or by adding manually), VQManager will hence forth store and maintain the endpoint details information for the calls participate by those Phone IDs. This imported endpoints will help to collect reports for specified endpoints rather than maintaining huge number of endpoints that may belonged to other network. It is highly recommended to import the desired set of phones to be monitored, this will help in better performance of VQManager application.
Please see the help link for configuration information:http://www.manageengine.com/products/vqmanager/help/working/endpoints.html#Managing_Enpoints
NOTE: Importing the Endpoints using this option will delete all the Endpoints discovered using auto-discovery and only the imported Endpoints will be listed for Endpoints reporting. Also auto-discovery of Endpoints will be disabled.

4j. Issue while reinitializing VQManager Database

Cause: Reinitialize script/batch file is to be invoked only when the server is not running. At times, a lock file named .lock gets created under <VQManager_Home>/bin directory. This creates problems when reinitializing the server even when it is not running.
Solution: Make sure you are not attempting to reinitialize while the server is running. Navigate to <VQManager_Home>/bin directory and check if ".lock" file had been created. If so, remove it and try again

Note: Reinitializing the database will remove the data completely from database. Hence we will not recommend to use the reinitialize script. Also taking the DB backup before reinitializing the DB will help you to restore the data if needed.

5. Installation Miscellaneous

5a. Having GCC libraries incompatibility problems while installing VQManager in Linux distributions

VQManager 6.2 installation requires the standard C++ libraries (libstdc++) and GCC (4.0 or above) core libraries (libgcc). The following list contains the minimal Linux distributions that contain these libraries.

Red Hat Enterprise Linux 4.4

Mandriva 2006

Debian 4.0

Ubuntu 5.10

CentOS 5

SuSE Linux 10

Fedora Core 4

If you have the above mentioned or higher versions of Linux distribution and still have problems installing VQManager, kindly check if the libraries have been installed. You can confirm it by checking the presence of libstdc++.so.6 under the /usr/lib directory and libgcc_s.so.1 under the /lib directory. If they are not available, kindly install the libgcc and libstdc++ libraries from your Linux distribution CD.

If you have a lower version of Linux distribution and still wish to run VQManager, you need to compile and install GCC 4.0 or above. Follow the below mentioned steps to install GCC.

Download the GCC Core and GCC g++ version 4.0 or above from http://ftp.gnu.org/gnu/gcc and extract it to, lets say GCC_SOURCE_DIR.

Create two directories for building and installing GCC: mkdir GCC_4_0_INSTALL ; mkdir GCC_4_0_COMPILE

Move to the compile directory: cd GCC_4_0_COMPILE

Invoke the configure script: ./configure --prefix=

After configuring, compile and install the GCC using the command "make && make install"

Run the command: echo ' /lib' >> /etc/ld.so.conf

Run /sbin/ldconfig to add the GCC install directory to the Linux library path.

Install latest VQManager version from the http://www.manageengine.com/products/vqmanager/download.html

Still you are facing the same issue, Check the log files available under <VQManager_Home>/logs directory. If you find any exceptions, send the log files to VQManager support. Also share us the GCC log files, you can find the files in /tmp/gcc_versionout***.txt. This will help us to analyze the exact cause of the issue.

5b. Issue while running as service in Ubuntu machine

Solution: We have fixed few issues related to installing VQManager as service while running in Ubuntu machine. Please follow the below steps to apply the fix for proper functioning of VQManager as a service.

Please download the fix from the below link.

http://www.manageengine.com/products/vqmanager/setup-service.sh

Save the setup-service.sh file into that directory.

Try installing VQManager as a service using command: sh setup-service.sh install.

Take a backup of setup-service.sh file present under /bin directory.

Also, please refer the below link http://manageengine.adventnet.com/products/vqmanager /help/getting_started.html#service_linux for installing VQManager as a service.

5c. Issue while installing VQManager in 64 bit linux

VQManager works well with 64 bit linux machine. However, for a 64 bit machine, it may be required to install 32-bit libraries for VQManager to work properly. So, kindly install the 32-bit libraries and check if the problem is solved.

Please refer the following documents for more details.

If the issue still persists, please upload the log files via 'Create' link in the 'Support Tab' of VQManager web-client for our further analysis.

5d. Failure in Server-startup

Cause:

During the previous run, if you had terminated the server abruptly or there was an unclean shutdown, some of the server processes would not have been terminated and the MySQL server instance would continue to run in the system.

Solution:

Forcefully terminate the MySQL Server instance (mysqld-nt.exe in Windows, mysqld in Linux). Try starting VQManager server again.

5e. Does VQManager support Windows2008 machine ?

At present VQManager does not support Windows Server 2008. VQManager uses WinPcap libraries for fetching the data from the interface card. The Windows Server 2008 support has been added in the beta version of WinPcap libraries. It is having some issue in reading the information from NIC and in-turn stops the call processing (data collection) thread of VQManager.

Workaround:

Hence, we request you to install VQManager in Windows 2000/2003/XP/NT/Vista machines or some other Linux flavors mentioned in our Systems Requirement page and the link is given below.
http://www.manageengine.com/products/vqmanager/system-requirements.html#OS

NOTE : Please be informed that standard stable WinPCap version is needed by VQManager. Please refer the below link for more information.

5f. 'npptools.dll not found' message while installing VQManager

Cause:

The "npptools.dll not found" issue is caused due to non-availability of WinPcap library files in your system.

Solution:
- WinPcap is the pre-requisite for installing VQManager. Please install the standard stable WinPcap in your machine from http://www.winpcap.org/archive/4.1.1-WinPcap.exe . After installing the WinPcap libraries, please reboot the VQManager installed machine and then try starting the VQManager server.

5g. Is there any workaround to make VQManager work properly in Windows Vista?

Procedure : To enable VQManager to work properly in the Windows Vista machine, the Winpcap needs to be manually installed. This package needs to be installed after the VQManager installation so that the dlls are updated properly.

5h. After uninstalling the product in windows, some folders are not getting deleted, why?

Cause: This usually happens when you try to uninstall the product immediately after shutting down the server.

Solution: Ensure that you uninstall the product only after the MySQL Server instance (mysqld-nt.exe process in Windows Task Manager/ mysqld in Linux) has been terminated completely after the server shutdown. Folders created at runtime, like ScheduleReport and user created folders will not be removed after un-installation.

5i. Unable to access VQManager Server through the Web Interface(http://localhost:8647) Why?

Cause #1 & Solution

Incomplete server start-up
Ensure that the server has successfully started. This can be verified by the presence of the message "Connect to: [http://localhost:8647 ]" in the console. Connect to the web client after seeing this message.

Cause #2 & Solution

Wrong URL
VQManager server and the Web Interface communicate through http. So ensure the URL contains HTTP.
http://<vqmanager_server_machine_name>:<port>/ For e.g. http://localhost:8647/

Cause #3 & Solution

Do you see any "FAILED" message in the Server Console?.
Check the log files available under <VQManager_Home>/logs directory. If you find any errors, send the log files to VQManager support

5j. Installation fails

Installation fails in Windows machine

Please check if you have administrator privileges to operate your Windows machine.
Take a screenshot of the error message that appears in the console when the installation fails.
Also, zip the ' logs ' directory. (By default it will be located at " C:ManageEngineVQManagerlogs ")
E-mail the screenshot and the zipped 'logs ' directory and also the complete details of the system configuration to VQManager support

Installation fails in Linux machine :

5k. Silent installation of VQManager

Procedure for silent installation of VQManager in Windows and Linux is given below.

In Windows:

To install VQManager in silent mode in Windows, first a response file(.iss) needs to be generated by installing VQManager in GUI mode which stores the required options for installation.
Procedure to create the response file:
A normal(GUI) setup receives the necessary input from the user in the form of responses to dialog boxes. However, a silent setup does not prompt the user for input. A silent setup must gets its user input from a different source, which is nothing but the InstallShield Silent response file(.iss).

To create a response file, open command prompt and browse to the location where you have stored the ManageEngine_VQManager.exe file and execute the following command.

ManageEngine_VQManager.exe -a -r -f1<response file path>

Where <response file path> is the path of the response file, such that the directory of the mentioned path should exist.
Please note that there should be no space char in between the option -f1 and the <responsefilepath>.
Ex: ManageEngine_VQManager.exe -a -r -f1C:VQManagerinstall.iss

Please ensure that the directory C:VQManager exists.

Response file usage:
Steps to use the response file to run the installer in silent mode.

Create the response file as said above.
Edit the response file if you want to specify different options than when it is created.
Run the following command

ManageEngine_VQManager.exe -s -a -s -f1<response file path>
Ex: ManageEngine_VQManager.exe -s -a -s -f1C:VQManagerinstall.iss

Now VQManager will be installed in the corresponding directory mentioned in the reference file. To change some fields like VQManager installation directory, web server port, DB port etc, you can edit the reference file with corresponding values and then use the above procedure.

In Linux:

For silent installation in Linux, please download the silent.txt file and then edit the file to fill the corresponding fields as required by you like VQManager installation directory, web server port, DB port etc in the place of <value> in the file and then use the below command after browsing to the bin file in the terminal. Make sure that you are executing the below command with root privileges.

<bin file path>#./ManageEngine_VQManager.bin -silent -options <Full path of txt file>

Ex: ./ManageEngine_VQManager.bin -silent -options /home/voip/silent.txt

Now VQManager will be installed in the corresponding directory mentioned by you in the .txt file.

6. Licensing Queries

6a. Unable to view some users Calls - Licensing/ Port-Mirroring

Licensing

If you have purchased Licensed version of VQManager, few calls may not be monitored if license limit is exceeded.
1. Enterprise - IP Phone based License
  - Please check with 'Monitored Endpoints' in Admin tab, if the number of IP Phone count has reached the licensing count. Then the calls participated by Endpoints not listed in the Monitored list may get dropped.
  - Solution: Please upgrade the license or delete/edit few unwanted Endpoints from the list
2. Service Provider - Concurrent Calls based License
  - If the Concurrent calls reached the licensing count, you will receive an alert message above the VQManager tabs.
  - Solution: Upgrade the license count, or else check if there is any stale active calls in Calls tab - If so please move them to 'Unmonitored' list. For getting more insight into channel/trunk utilization, please refer the 'Concurrent Calls' link available in 'Report tab'
3. Free version - On trail expires
  - Trail version of VQManager will get expire on completion of 30 days from installation. After which, it will be moved to Free version, where you can monitor 10 IP phones only. Please check this from the message available in the VQManager web-client.
  - Solution: Request an extended trail license from VQManager support.

NOTE: Monitored Endpoints link under the Admin Tab will be available for IP Phone based licensing and for Free versions. It will not be available for Service Providers - Concurrent calls based licensing and also for trail versions.

6b. There is no link for 'Monitored Endpoints' in the Admin tab in VQManager trial edition

Cause & Solution: In the trial edition, there is NO limitation on the number of IP Phones to be monitored. You can monitor VoIP calls from all your IP Phones. Therefore, the 'Monitored Endpoints' link does not appear when the product is run in trial version mode. Once, the trial version expires, you will have an option to choose the IP Phones to be monitored. See "Monitored Endpoints" section in the Help Document for more details.

7. User Management

7a. Can I delete the Administrator account in user management?

Caution : Under the User management of VQManager, there will be two default users named 'admin' and 'guest'. For proper functioning of VQManager user management, the 'admin' account with 'administrator' privilege should not be deleted. Just in case you have deleted accidentally, please contact VQManager support for assistance.

7b. Password reset options - Running DB Query

Questions :

Have you tried changing the default password and forget the new password during the login?
While login, does the status says 'Invalid password' message?

Solutions :

Contact VQManager support for password reset.

8. Miscellaneous

8a. Created 'Custom Report' shows no data and the email received also does not contain data

Reason : Once on creating the custom reports, the following message will be shown "Report is scheduled for data collection. Please view after some time". As stated, by default the reports will be generated after 15 mins(max). Hence we recommend you to chose the Schedule time at-least 15mins in advance from the time of report creation. This will help you to the view the call list matching your custom criteria.

8b. Repeated Invalid Username/Password, Session expires

Possible Reason : If you notice repeated 'Invalid Username/password' message in the VQManager login page while provide correct credentials. One possible error could be VQManager application could have lost its connection with the database(MySQL), hence please restart the VQManager server and try logging in, this could solve the issues

8c. My Web Interface looks crippled

Cause #1 & Solution

Incompatible Browser
Refer to the System Requirements, and check whether your browser is supported.

Cause #2 & Solution

JavaScript not enabled
JavaScript has to be enabled in your browser for you to work with the Web Interface.

8d. How can I find my Build number?

Always mention Build Number while contacting VQManager support

8d. How can I know the trial version expire period?

Information Base

Port Mirroring

Problem with Port-mirroring/SPAN in the network. VQManager monitors the VoIP calls by sniffing the VoIP packets both Signaling(SIP, SCCP & H.323) & Media(RTP,RTCP) in the network. In a switched network, by default, it is not possible to get all the packets flowing in a network from one machine. To overcome this, port mirroring/SPAN has to be enabled in the switch to send all the VoIP packets to the machine where VQManager is running. Alternatively, for smaller networks Hubs may also work. Network Taps are another workable solution to route multiple interface traffic to VQManager listening NIC. For proper working of VQManager, both Media and Signaling traffic has to be directed to VQManager server.

For information on configuring port-mirroring in a number of managed switches like 3Com, Cisco Systems, Linksys, NETGEAR, Avaya etc., please check: http://wiki.wireshark.org/SwitchReference/

NOTE: While spanning the Voice traffic to VQManager, please make sure that there is no chance for packet duplication. Check whether you have mirrored Ingress/Egress/Both traffic to VQManager listening NIC

Protocol Support

Call Signaling Protocol :

VQManager Sniffer supports parsing Signaling protocol such as SIP[Session Initiation Protocol]-(RFC 3261), H.323, Cisco SCCP (Skinny) and the media protocol such as RTP/RTCP(RFC 3550). VQManager does not support SIP with TCP, or any other protocols. If you have a hybrid protocol environment and if calls are not getting monitored, then contact VQManager support with packet capture of the problematic calls scenarios captured using some packet capturing tools such as Wireshark [HOW TO GET PCAP] from the VQManager listening interface and detailed description on your network.

Registration processing Protocols :

VQManager Sniffer supports parsing Signaling protocol such as SIP[Session Initiation Protocol] Registration packet processing alone. If you face any issues related to SIP Registration packet processing after checking this, VQManager support with packet capture of the registration packets captured using some packet capturing tools such as Wireshark [ HOW TO GET PCAP] from the VQManager listening interface and detailed description on your network.

Unmonitored Calls

This may occur in case of complete call signaling packets(Call initiation and termination traffic) not reaching VQManager sniffer listening interface (NIC). It may be due to improper port-mirroring/SPAN or installing VQManager in a wrong place.

And VQManager by default check if there exists an interruption of Call signaling traffic for 60 seconds and end the call stating Unmonitor due to no media transmission with Call status 8003. User can increase this default check from 60 seconds or even disable this configuration by configuring 0 seconds in Sniffer configuration advanced settings page.
This link provides the list of Custom Codes used in VQManager.

Procedure to get packet capture for VQManager support

Option 1: From VQManager web-client itself

In VQManager webclient, under the 'Admin' tab --> 'Proxy Settings', make sure that your proxy server settings are configured correctly.
Under the 'Support' tab, click on 'Create' button in the support-file creation.
Enter the details in the support form window that opens up.
Check the 'Add packet capture file to support log information' option. By default, the interface which VQManager's sniffer is currently listening to, will be chosen(make sure the same)
For 'Time of Capturing' choose a value from '30 secs' to '15 mins' from the dropdown menu and for the 'Maximum of' packets option, choose a value from '5000' and '500000' packets from the dropdown menu.The default filter string will be "tcp || udp || vlan".
Support File Creation and sending process will take time depending on the load on the server and the size of the log files. Please do not close the support form window till the process is complete.

Option 2: Using some packet capturing tools such as wireshark

From the VQManager listening interface using a packet capturing tools such as Wireshark gets us the packet capture. Make sure you have initiated few calls during the capture and also check for the presence of VoIP calls in Wireshark using 'Telephony/Statistics->VoIP calls' link in the top Tool bar. Upload the captured file.

Procedure for uploading the captured log files :

Please share with us the uploaded packet capture or files using the below link

Advanced QoS Calculation Logic in VQManager

VQManager reports on five quality parameters such as Delay, Jitter, Packet Loss, MOS and R-Factor

Mean Opinion Score(MOS) and R Factor Calculation :
MOS & R-Factor are calculated using a formula suggested by E-Model, ITU-T [Rec. G.107]. Please refer for detailed information in calculation logic and formula http://www.itu.int/ITU-T/studygroups/com12/emodelv1/tut.htm MOS and R-Factor are derived from a formula which takes into account of various factor right from Latency, Loss etc,. to Codec used, Echo etc,. For few parameters, E-Model provides the default value based on Codec used. They arrive on this default value by running large number of test calls in different environment.

MOS is measured in the scale from 1 to 5. Where 1 is poor and 5 is excellent. Likewise R-Factor ranges between 1-100, 1 is poor and 100 is Excellent. As MOS and R-Factor where derived from same formula, both these values will share the same trend(spikes and variations).

Quality Scale used in VQManager

Category
MOS
R-Factor
Good >= 3.6
>= 70
Tolerable Between 3.1 and 3.6
Between 60 and 70
Poor <= 3.1
<= 60

Note 1: The above set scale is configuration by clicking "Configure" link in the MonitorTab-> 'Voice Quality' box

Note 2: Any Impact on Packet loss may bring down the MOS adversely.

R-Factor measuring Scale suggested by E-model

Range of E-model Rating R
Speech transmission quality category
User satisfaction
90 £ R < 100
Best
Very satisfied
80 £ R < 90
High
Satisfied
70 £ R < 80
Medium
Some users dissatisfied
60 £ R < 70
Low
Many users dissatisfied
50 £ R < 60
Poor
Nearly all users dissatisfied
NOTE 1 – Connections with E-model Ratings R below 50 are not recommended.

Delay and Jitter Calculation
Delay is calculated from the RTCP packet and in the absence of RTCP packets, the delay is calculated from the RTP packets. Please refer RTP RFC http://www.faqs.org/rfcs/rfc3550.html for detailed flow of Delay & Jitter Calculation formula and explanations.

About E-Model ITU-T

The E-model (ITU-T Rec. G.107 [1]) is a transmission planning tool that provides a prediction of the expected voice quality, as perceived by a typical telephone user, for a complete end-to-end (i.e. mouth-to-ear) telephone connection under conversational conditions. The E-model takes into account a wide range of telephony-band impairments, in particular the impairment due to low bit-rate coding devices and one-way delay, as well as the "classical" telephony impairments of loss, noise and echo. The E-model is based on modeling the results from a large number of subjective tests done in the past on a wide range of transmission parameters.

How to take packet capture using Wireshark and analyze the VoIP calls

Steps for packet capturing in Wireshark:

Click on "Capture --> Options" in tool bar as shown in above screenshot.
Select the proper interface, provide the filter string and then click 'Start'.
After clicking 'Start', the packet capturing will start. Capture the packets until some desired time interval.
Now click on 'Capture --> Stop'. Packet capturing will stop.
Now click on 'File --> Save As' to save the packet file.

Analyzing packet capture file in Wireshark:

Open the packet capture file using WIreshark.
In Windows, we can double click on the packet capture file(If Wireshark is installed) to open it in Wirehsark.
In Linux, Go to terminal and type "wireshark ".
Now see the presence of VoIP calls in it, click on "Statistics/Telephony --> VoIP calls". A new window will open which gives the list of VoIP calls present, if any.
Now to see the Call Flow diagram for a call, select the call from the list and click on 'Graph' option present at the bottom of the window. We will be able to see the graph.
To see the QoS parameters like Delay, Jitter etc for a call in Wireshark, select an RTP packet for that call(We can select the RTP packet for a call from the graph, it will be automatically selected in the main window also) and then click 'Statistics/Telephony --> RTP --> Stream Analysis'.

VQManager Best Practices Guide

The document describes some best practices for VQManager in the following areas
   1. Hardware and Software Requirements
        1. Server
        2. Client
        3. Software requirements
   2. Configuring VQManager Performance
         1. MySQL Tuning
         2. Packet Process Tuning
         3. Storing Raw packets
         4. Import Endpoints option
         5. Running VQManager in Multi CPU Machine
         6. Java(JVM) Memory Tuning
   3. VQManager Database Management
         1. Database Clean up
         2. Database Back up
   4. Other General Guidelines
         1. Deployment Scenarios
1. Hardware and Software Requirements
The minimum hardware requirements for installing and working with VQManager are given below:
a. Server:
      1.8 GHz Pentium IV processor or equivalent
      1 GB of RAM
      2 GB of disk space
Note: The above mentioned requirements are valid upto 100 simultaneous calls. The Optimal hardware
configurations required depends on the number of simultaneous calls that happen in the network. The
following table recommends the RAM size requirements for different ranges of simultaneous calls.
No. of simultaneous calls Processor speed             RAM
0-100                      1.8 GHz                     1 GB
101-300                    2 GHz                       2 GB
301-500                    2 GHz , Dual core           4 GB
501-800                    2 GHz, Quad core            4 GB
800-1000                   4 GHz, Quad core            8 GB
Disk Usage: The rough disk usage of VQManager is 100 MB for every 10,000 calls stored.
b. Client:
As VQManager has a web client, all that is needed is a browser to connect to it. VQManager supports Internet
Explorer 6.0 and later, and Firefox 2.5 and later. The recommended monitor resolution is 1024 x 768.
c. Software Requirements:
Recommended software requirements for installing and running the VQManager are given below
Supported Operating Systems

Windows 2000 / 2003 / 2008 /2008 R2
Windows 7 / XP / NT / Vista

The following Linux flavours contains the minimum configuration required for installing VQManager

Red Hat Enterprise Linux 4.4
Mandriva 2006
Debian 4.0
Ubuntu 5.10
CentOS 5
SuSE Linux 10
Fedora Core 4

Supported Browsers
Internet Explorer 6.0 and later
Firefox 1.5 and later

To work with lower Linux distributions, you will need to install GCC 4.0 or above. See the <LINK>GCC
incompatibilities<LINK>
for smooth installation of VQManager in other lower Linux distribution
2.Configuring VQManager - Performance Tuning
a. MySQL Tuning
The default database parameters in VQManager are tuned to handle 100 concurrent calls in a server having 1
GB RAM. If your network has more than 100 concurrent calls and the VQManager installed machine contains more RAM, then you
can tune MySQL parameters to extract better performance . This will significantly improve the Client and
Server performance of the VQManager. We have bundled tuned database parameters for different RAM sizes.
You can replace the appropriate file according to avaiable RAM in VQManager installed machine. Please follow
the below procedure to tune the database
      Shutdown the VQManager server if it is running.
      Go to <VQManager_Home>/conf directory.(By default <VQManager_Home> is C:ManageEngineVQManager for Windows and /root/ManageEngine/VQManager for Linux)
      Please take a backup of mysqlConf.txt file.
      If you use 4 GB RAM then please copy mysqlConf_4GB.txt file to mysqlConf.txt. Please choose mysqlConf_16GB.txt or
      mysqlConf_2GB.txt files if you use 16 GB or 2 GB RAM respectively and copy it to mysqlConf.txt
b. Packet process tuning
VQManager temporarily stores the sniffed packets and processed data in series of memory buffers and files
before it is persisted to database. You can tune number and size of memory buffers and files used in call
processing according to the concurrent calls in your environment.
Please change below parameters in <VQManager_Home>/conf/VoIP/Sniffer.conf file according to number of concurrent calls.

Parameter                 0-100             101-300       301-500           501-800          801-1000
                           Concurrent        Concurrent    Concurrent        Concurrent       Concurrent
                           Calls             Calls         Calls             Calls            Calls

dataBufferCount             5                 8             10                12               15

dataBufferSize              1                 2             4                 8                10

registrationsBufferCount    5                 8             10                12               15

registrationsBufferSize     1                 2             4                 8                10

vqDataFileMaxCount          10                10            12                15               20

vqDataFileMaxSize           10                12            15                20               25

kernalBufferSize*           2                 3             4                 7                10

Note : kernalBufferSize parameter only applicable to windows. It reduces the dropping of packets
in kernal level.
b. Storing Raw Packets
You can disable this option in VQManager Client->Admin->Sniffer, if you do not need to store raw
signaling packets. This will reduce the VQManager disk consumption significantly.
c. Import Endpoints option
Import Endpoints option in Endpoints tab allows user to limit the Endpoints to be listed in Endpoints tab. In
Concurrent calls license and trial version, VQManager stores all endpoints participated in all the calls. In huge
concurrent call cases, this results in large number of endpoints are reported which does not belong to user network (For
example endpoints of outside networks and public numbers) will be maintained in VQManager. You can avoid
this by importing the endpoints to be monitored. Once endpoints list imported, VQManager lists and provide
Endpoint reports only for the imported Enpoints. You can get more information on this in the below help
document link.
<LINK>http://www.manageengine.com/products/vqmanager/help/working/endpoints.html<LINK>
d. Running VQManager in Multi CPU Machine
When running VQManager in a multi-CPU machine with more than 2 GB RAM, the JVM will use the parallel
garbage collector (for cleaning up the free objects). This may degrade the performance of VQManager in huge
traffic networks. If you face a high CPU usage by VQManager, Please add "-XX:+UseSerialGC" property
before " -Xmx150M " in the <VQManager_Home>/bin/run.bat(run.sh in case of Linux) to force the JVM
to choose serial Garbage Collector.
e. Java(JVM) Memory Tuning
VQManager Java(JVM) heap size can be increased according to your environment by editing the -Xmx
parameter in the following files
<VQManager_Home>/bin/run.sh for Linux
<VQManager_Home>/bin/run.bat for Windows
VQManager installed as service
On such cases memory tuning can be achieved by editing the following parameters in wrapper.conf file
available under the directory
<VQManager_Home>/conf/
wrapper.java.initmemory – Initial Java Heap Size
wrapper.java.maxmemory – Maximum Java Heap size
Recommended setting for JVM Heap size various RAM sizes and assuming VQManager is running on a
dedicated server.
RAM Size                 JVM Heap Size
1 GB                     150M
2 GB                     300M
4 GB                     600M
8 GB and Above           1024M
3. VQManager Database Management
a. Database Clean up
It is advisable to have only the required period of history data in VQManager. This will reduce the hard disk
space used by the VQManager and also improve the client performance. By default VQManager is configured
to delete call data older than 1 week and registrations data older than 3 days by running a schedule everyday
at 1:00 am(Low traffic hours). Since the delete operation is time consuming, it is advisable to set the clean
up policy to run in the off-peak timings. Please refer the below document for changing the settings of
Database Clean Up
<LINK>http://manageengine.adventnet.com/products/vqmanager/help/working
/admin_operations.html#__database_administration<LINK>
b. Database Back up
It is very important to backup the VQManager data in database every week, so that data is not lost in case of
any disaster. Please follow the below steps to take a database backup
    1. Shutdown the VQManager server.
    2. Go to <VQManager_Home>/bin directory
    3. Execute the command backupDB.bat (backupDB.sh in case of Linux)
    4. This creates backup of the database in a zip file under the <VQManager_Home>/Backup directory
4. Other General Guidelines
a. Deployment Scenarios
VQManager has two data input options.
    1. Sniffer
    2. CDR
For data inputs and deployment scenarios please refer the below document
<LINK>http://manageengine.adventnet.com/products/vqmanager/voip-monitoring-tool-vqmanager-deployment.html<LINK>

Distributed Edition Trouble Shooting

Probe server is not starting(When started for the first time)

Here are some of the possible reasons

See if the Central server is up and running if you are starting the Probe server for the first time.
Make sure that the Central server installed machine is reachable from the Probe server installed machine.
Make sure that the Central server details provided are correct. You can check the details in the server.properties file present under <VQManager_Home>conf directory.

If you are still facing the problem, please zip the log files present under the <VQManager>/logs directory and share them with us at vqmanager-support[at]manageegnine.com for our analysis with a brief description of your problem.

Created a Site but it is not listed in the site filter?

A Site will be listed in the site filter/dropdown only if the probe server is associated to the site. Please check if you have associated the probe to a particular/any Site. Please click on the "+/-" symbol present beside the Site name to associate the Probe to the Site.

Created a Site but not able to see any data in the Central server Dashboard

The data for a Site will be shown in the Central dashboard only if the Site is associated with the corresponding Probe server. Please click on the "+/-" symbol present beside the Site name to associate the Probe to the Site. If you are not able to see the data even after associating the probe, please check the following.

Please make sure that you have configured the sniffer in the probe associated for the corresponding site.
If the sniffer is configured and still not able to see the data, please check the port-mirroring to see if all the traffic is port-mirrored to the VQManager listening interface. You can check the traffic coming to the interfaces in the Sniffer configuration wizard.

How to change/configure probe data sync up time?

You can find the list of all the probes under "Admin tab --> Probe". Here you can find an option to edit (note pad icon) the probe details under "Action" header. Click on that. Here you can do all the possible configuration changes for the probe server including the data sync up time, host name, web port etc.

Data mismatch in Central and Probe servers

This is a known issue. Consider a case, If you are storing 2 weeks of data in the Central server and only one week data in the Probe server, then the you can see the two weeks data in the Central server. If you further to drill down to the Probe sever to see the call details, you will not find the data as it is not present in the Probe server. For this case, please configure same values/days as datacleanup duration in Central and Probe servers.

I am not able to enable the probe server. It is disabled.

Please check if your license has expired. Once the license is expired, the probe will be disabled automatically and cannot be enabled. Please apply the license if you have already received it or contact the VQManager support for assistance. After applying the license, using edit option of probe enable the probe to collect data.

I do not want to monitor a Site. How can I do that.

If you wish to unmonitor a Site temporarily, you can do it by disabling the corresponding probe server associated to that Site. You can start monitoring the Site again by enabling the probe. All the historical data will be retained, based on the data cleanup policy, even if you disable the probe.

If you do not want a Site to be monitored, you can delete it after un-associating the probe server associated, if any. But all the calls data corresponding to that Site will be deleted if a Site is deleted and it cannot be restored.

How can I change the Probe web server port and host name?

I am not receiving any emails of the reports that I have configured in VQManager.

Please make sure that you have provided the proper mail server setting in VQManager probe and central. You can check it by going to "Admin tab --> Mail Settings" of that corresponding server. You can also check the mail settings provided by clicking on the "Test" link available on the right side on the same page.

Not able to launch the probe server UI in the Central server even though the probe server is up and running

Please check if the probe server installed machine is reachable from the Central server installed machine. If the Central and probe servers are in two geographically distinct locations and have any proxy sever, make sure to provide the proxy server settings in the Central server under the admin tab.

Comments (4)

Attachments (8)

RSS of this page