Azure-quickstart-templates: Apllication Gateway/GoAccess: Invalid URI: The URI scheme is not valid (SAS URL)

[marcelboccato]

After deploying the GoAccess solution to ApplicationGateway, I checked the log:

/var/log/azure/Microsoft.Azure.Networking.ApplicationGateway.LogProcessor/application.log

2019-01-03 17:39:55,941 ERROR [6] ?.? - RefreshBlobs: Caught Exception: System.AggregateException: One or more errors occurred. (One of the query parameters specified in the request URI is not supported.) ---> Microsoft.WindowsAzure.Storage.StorageException: One of the query parameters specified in the request URI is not supported.
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternalT
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Networking.ApplicationGateway.LogProcessor.LogBlobFetcher.RefreshBlobs(Object state, ElapsedEventArgs e) in E:\src\networkmonitoring\Tools\src\AppGatewayLogProcessor\Services\LogBlobFetcher.cs:line 282

---

This error refers to the contents of the "blobsasuri.key" file in the same directory. Inside, I've inserted the "Blob service SAS URL", that works fine if copy/paste on browser, listing all containers from my storage account.

What is the correct format for inserting the URI on file "blobsasuri.key"? Could be a bug from ApplicationGateway.LogProcessor?

Thanks!

[subsarma]

We are investigating this, will respond asap.

[frankfuu]

I'm also experiencing the same issue.

[frankfuu]

Hi @marcelboccato , did you have a temporary work around for this?

[hoodbsa]

I've mitigated this issue by using container-level SAS URL (for container named "insights-logs-applicationgatewayaccesslog") instead of account-level SAS URL.

[frankfuu]

Thank you so much @hoodbsa! amazing! finally got it working!

Here was what I did after ssh'ing into the provisioned vm

# stop services
systemctl stop appgatewaylogprocessor
systemctl stop goaccess

# backup original blobssasuri.key and update it to container-level sas URL
sudo cp /usr/share/appgatewaylogprocessor/blobsasuri.key /usr/share/appgatewaylogprocessor/blobsasuri.key.bak
sudo echo "https://<accountname>.blob.core.windows.net/insights-logs-applicationgatewayaccesslog/?sv=<your SAS TOKEN>" | sudo tee --append /usr/share/appgatewaylogprocessor/blobsasuri.key

# restart services
systemctl start appgatewaylogprocessor
systemctl start goaccess

# optionally check if appgatewaylogprocessor is processing correctly
tail -f /var/log/azure/Microsoft.Azure.Networking.ApplicationGateway.LogProcessor/application.log

# optionally check service statuses
sudo systemctl status appgatewaylogprocessor
sudo systemctl status goaccess

[marcelboccato]

@hoodbsa @frankfuu Thank you SO MUCH for this! Worked flawless! Helped me a lot!!

One last doubt: I have two Appgateway, one for Stagging puorposes and one on Production environment. GoAccess is fetching data from wich one? Both?

Is there a way to retrieve data from just one app gateway?

[frankfuu]

@marcelboccato, not too sure but you could probably send each of the app gateway logs to different storage accounts for each environment then ask goaccess to fetch only the logs for the production environment.

[anshul291995]

@marcelboccato You can specify the name of application gateway while deploying the template on azure if you have multiple application gateways pumping logs to same storage account. Alternatively the suggestion from @frankfuu works too.

[marcelboccato]

@anshul291995 @frankfuu i figure out that simple inserting the name from my Appgateway on file "appgwlogsbloburlregex" worked. But now Goaccess can't start. I've got an error:

Parsed 1 linesproducing the following errors:

Token '▒▒"2019-02-13' doesn't match specifier '%d'

Format Errors - Verify your log/date/time format

---

Can you post the content from "/usr/share/appgatewaylogprocessor/files/scripts/launch_goaccess.sh" to compare?

Mine is:
/usr/local/bin/goaccess /var/log/azure/Microsoft.Azure.Networking.ApplicationGateway.LogProcessor/access.log* -o /var/www/html/report.html --real-time-html --port=8080 --log-format='"%dT%tZ"{%^:"%h",%^:"%m",%^:"%U",%^:"%q",%^:"%u",%^:"%s",%^:"%H",%^:"%b",%^:"%T",%^:%v}' --time-format='%T' --date-format='%Y-%m-%d'

Thank you!

[frankfuu]

hi @marcelboccato,

Mine was
/usr/local/bin/goaccess /var/log/azure/Microsoft.Azure.Networking.ApplicationGateway.LogProcessor/access.log* -o /var/www/html/report.html --real-time-html --port=8080 --log-format='"%dT%tZ"{%^:"%h",%^:"%m",%^:"%U",%^:"%q",%^:"%u",%^:"%s",%^:"%H",%^:"%b",%^:"%T",%^:%v}' --time-format='%T' --date-format='%Y-%m-%d'

Which is found in /files/scripts/launch_goaccess.sh which is part of a zip file named AppGatewayLogProcessor that gets unzipped and installed when the this arm template gets installed.