mirror of
https://github.com/clearlinux/telemetrics-backend.git
synced 2026-09-05 05:11:56 +00:00
Redesigned the telemetry backend app.
- Added migration code to modify 'records' table as follows:
- Rename 'os_name' ==> 'system_name'
- Rename 'machine' ==> 'host_type'
- Rename 'payload_format_version' ==> 'payload_version'
- Rename 'record_format_version' ==> 'record_version'
- Rename 'tsp' ==> 'timestamp_client'
- Rename 'tsp_server' ==> 'timestamp_server'
- Modify 'payload' type from binary to text
- Remove 'buildstamp'
- Remove 'dupe_of'
- Remove 'dupecount'
- Remove 'dupemaster'
- Remove 'security'
- Remove 'hide'
- Remove 'icon'
- Add 'build'
- Add 'classification'
- Updated code to reflect the above design changes.
- Updated code to remove references to 'backtrace' column.
But not from the table because data migration from
'backtrace' to 'payload' should be done first.
- Added migration code to update the table data based on the above
design changes.
- Added migration code to update the Views based on the above design
changes.
- Removed the 'collector' app.
- Modified database name to 'telemetry'.
- Added code to filter out build id >= 100000.
This commit is contained in:
Reagan Lopez
@@ -1,3 +1,4 @@
|
||||
*.pyc
|
||||
*.log
|
||||
*.swp
|
||||
*.idea
|
||||
|
||||
-11
@@ -1,11 +0,0 @@
|
||||
language: python
|
||||
python:
|
||||
3.5.2
|
||||
install:
|
||||
pip3 install -r requirements.txt
|
||||
script:
|
||||
make run-tests
|
||||
services:
|
||||
postgresql
|
||||
before_script:
|
||||
psql -c 'create database testdb;' -U postgres
|
||||
@@ -17,21 +17,10 @@ release:
|
||||
git tag -a -m "Release $(VERSION)" v$(VERSION); \
|
||||
printf "\nNew release $(VERSION) tagged!\n\n"
|
||||
|
||||
run-collector:
|
||||
@echo $(WARNING)
|
||||
FLASK_APP=collector/collector/report_handler.py flask run --host $(HOST) ${PORT}
|
||||
|
||||
run-telemetryui:
|
||||
@echo $(WARNING)
|
||||
FLASK_APP=telemetryui/telemetryui/views.py flask run --host $(HOST) ${PORT}
|
||||
|
||||
run-tests:
|
||||
PYTHONPATH=/$(shell pwd)/collector python3 collector/collector/tests/headers.py
|
||||
PYTHONPATH=/$(shell pwd)/collector python3 collector/collector/tests/api.py
|
||||
PYTHONPATH=/$(shell pwd)/collector python3 collector/collector/tests/purging.py
|
||||
PYTHONPATH=/$(shell pwd)/collector python3 collector/collector/tests/validation.py
|
||||
PYTHONPATH=/$(shell pwd)/collector python3 collector/collector/tests/parsers.py
|
||||
|
||||
tag-production:
|
||||
@TODAY=`date "+%F-%H%M%S"`; \
|
||||
git rev-parse production-$$TODAY &> /dev/null; \
|
||||
|
||||
@@ -7,8 +7,7 @@ This project provides the server-side component of a complete telemetrics
|
||||
client-side component source repository lives at
|
||||
https://github.com/clearlinux/telemetrics-client.
|
||||
|
||||
It consists of two Flask applications: an ingestion app, `collector`, for
|
||||
records received from telemetrics-client probes; and a web app, `telemetryui`,
|
||||
It consists of a Flask application `telemetryui`,
|
||||
that exposes several views to visualize the telemetry data. The `telemetryui`
|
||||
app also provides a REST API to perform queries on the data.
|
||||
|
||||
@@ -39,8 +38,8 @@ applications.
|
||||
To control access to the applications, it is recommended that system
|
||||
administrators leverage web server authentication.
|
||||
|
||||
Regarding alternate deployment scenarios, one might want to host the collector,
|
||||
telemetryui, and database on three separate servers/VMs; and implement network
|
||||
Regarding alternate deployment scenarios, one might want to host the telemetryui
|
||||
and database on three separate servers/VMs; and implement network
|
||||
access controls for these systems. The in-tree deployment script does not
|
||||
support these types of deployments, but with minimal modification to the
|
||||
source, they should be possible.
|
||||
@@ -99,7 +98,7 @@ $ ./deploy.sh -H localhost -d clr -a install
|
||||
During script execution, you will be prompted for user input:
|
||||
|
||||
* The first prompt begins with "DB password:", asking for a password to set for
|
||||
the `telemdb` database. If you do not enter a value before pressing ENTER,
|
||||
the `telemetry` database. If you do not enter a value before pressing ENTER,
|
||||
the password will be `postgres`.
|
||||
|
||||
* If your sudo access requires a password, you will be prompted for that
|
||||
@@ -125,32 +124,6 @@ working with. Its default value is the upstream repo location on
|
||||
github.com/clearlinux. The `-s` option lets you select a different git branch
|
||||
to install/deploy from rather than "master", the default value.
|
||||
|
||||
## `collector` operation
|
||||
|
||||
The `collector` app is an ingestion app that handles POST requests to `/`
|
||||
or `/v2/collector` at the web server location where telemetrics-backend has
|
||||
been installed. The requests are analyzed to make sure required header fields
|
||||
are present and that values are correct.
|
||||
|
||||
Generally, the POST requests are sent by probes from the telemetrics-client,
|
||||
since these probes use the client library, libtelemetry, to create well-formed
|
||||
records capable of being processed by telemetrics-backend.
|
||||
|
||||
At minimum, make sure that your telemetrics-client configuration in
|
||||
`/etc/telemetrics/telemetrics.conf` specifies the correct server URL for the
|
||||
`server` config option. For example, if telemetrics-backend is hosted at
|
||||
`example.com`, the client config should contain `server=http://example.com/` or
|
||||
`server=http://example.com/v2/collector`.
|
||||
|
||||
|
||||
## `collector` plugable payload parsers
|
||||
|
||||
By default `collector` will save the payload record from telemetry messages as is
|
||||
received, there are cases when is desirable to apply additional transformations to
|
||||
payload on selected classifications. This use case is covered by custom payload
|
||||
parsers/(or transformations). For more information on this feature read specific
|
||||
[documentation](/collector/collector/parsers/README.md)
|
||||
|
||||
## `telemetryui` views
|
||||
|
||||
The `telemetryui` app is a web app that exposes several views to visualize the
|
||||
@@ -158,7 +131,7 @@ telemetry data and also provides a REST API to perform queries on record data.
|
||||
|
||||
The current views are:
|
||||
|
||||
* Records view - a paginated list of all records in the `telemdb` database that
|
||||
* Records view - a paginated list of all records in the `telemetry` database that
|
||||
have been accepted by the `collector`. The records are presented in tabular
|
||||
format and the columns map to select fields from the `records` database table.
|
||||
At the top of the view, an HTML form can be used for "advanced searches",
|
||||
@@ -189,8 +162,7 @@ The current views are:
|
||||
reports are grouped by "guilties"; a guilty is a frame from a crash backtrace
|
||||
chosen as the best candidate for the cause of the crash. The logic for
|
||||
determining crash record guilty frames accepts user input; the user can
|
||||
identify which frames in a backtrace are never guilty. Guilty processing occurs
|
||||
asychronously in a uWSGI spooler process for the `collector` app.
|
||||
identify which frames in a backtrace are never guilty.
|
||||
|
||||
* MCE view - charts that display MCE (machine check exception) data from a
|
||||
patched version of `mcelog` that uses libtelemetry to create and send
|
||||
@@ -228,24 +200,6 @@ go to relevant documentation.
|
||||
|
||||
## Special configuration
|
||||
|
||||
### Configuring the `collector` TID
|
||||
|
||||
The `collector` requires a Telemetry ID (TID) header value set with the HTTP
|
||||
header `X-Telemetry-TID`. The telemetrics-client daemon adds this header to
|
||||
telemetry records, and its default value is used for identifying records from
|
||||
Clear Linux OS systems.
|
||||
|
||||
However, when you are deploying your own instance of telemetrics-backend and
|
||||
have deployed telemetrics-client to systems configured to send records to this
|
||||
instance, it is recommended to generate your own random TID (e.g. using the
|
||||
`uuidgen` program). Once you have generated a TID, the following steps are needed:
|
||||
|
||||
* Configure telemetrics-client to add this TID to records by modifying the
|
||||
`tidheader` value in `/etc/telemetrics/telemetrics.conf` on systems sending
|
||||
the telemetry data to your `collector`.
|
||||
* Configure your `collector` app to accept records with this TID by modifying
|
||||
`collector/collector/config.py`.
|
||||
|
||||
### Configuring nginx for TLS
|
||||
|
||||
The `sites_nginx.conf` config file is already enabled to accept TLS connections
|
||||
@@ -262,16 +216,6 @@ comment out TLS-related nginx configuration. Specifically, it will comment out
|
||||
the `listen 443 ssl`, `ssl_certificate`, `ssl_certificate_key`,
|
||||
`ssl_protocols`, and `ssl_ciphers` directives.
|
||||
|
||||
### Configuring record retention time
|
||||
|
||||
The `collector` app uses the uWSGI cron interface to purge records in the
|
||||
database on a daily basis that are older than a certain age. By default, a
|
||||
record will only be kept in the database for 5 weeks (starting from the
|
||||
timestamp the `collector` received the record). To modify the retention
|
||||
time, update the `MAX_DAYS_KEEP_UNFILTERED_RECORDS` value in
|
||||
`collector/collector/config.py` after installation, and restart
|
||||
uWSGI for the new setting to take effect.
|
||||
|
||||
## Using the REST API
|
||||
|
||||
A REST API for querying records is available at "/api/records". The API returns
|
||||
@@ -351,7 +295,7 @@ Database migrations are managed using
|
||||
[Flask-Migrate](https://flask-migrate.readthedocs.io/en/latest/). Upon initial
|
||||
install of telemetrics-backend, the first migration will be applied, and any
|
||||
additional migrations in the `telemetryui/migrations/versions/` directory will
|
||||
be applied in sequence and upgrade the `telemdb` schema to the latest version.
|
||||
be applied in sequence and upgrade the `telemetry` schema to the latest version.
|
||||
|
||||
To create a new migration, you can follow the steps below:
|
||||
|
||||
|
||||
@@ -1,47 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
from flask import Flask
|
||||
from . import config
|
||||
from flask_sqlalchemy import SQLAlchemy
|
||||
from logging.handlers import RotatingFileHandler
|
||||
|
||||
|
||||
def configure_app(config_object, app):
|
||||
app.config.from_object(config_object)
|
||||
db = SQLAlchemy(app)
|
||||
|
||||
app = Flask(__name__)
|
||||
db = SQLAlchemy()
|
||||
app.config.from_object(config.Config)
|
||||
|
||||
try:
|
||||
# try importing from the local dev configuration if it exists
|
||||
from . import config_local
|
||||
app.config.from_object(config_local.Config)
|
||||
except Exception as e:
|
||||
print(e)
|
||||
pass
|
||||
|
||||
from .model import *
|
||||
from . import report_handler
|
||||
|
||||
handler = RotatingFileHandler(app.config['LOG_FILE'], maxBytes=10000, backupCount=1)
|
||||
handler.setLevel(app.config['LOG_LEVEL'])
|
||||
app.logger.addHandler(handler)
|
||||
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
@@ -1,55 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import logging
|
||||
|
||||
|
||||
class Config(object):
|
||||
DEBUG = False
|
||||
TESTING = False
|
||||
LOG_LEVEL = logging.ERROR
|
||||
SQLALCHEMY_DATABASE_URI = 'postgresql://postgres:@@db_password@@@localhost/telemdb'
|
||||
SQLALCHEMY_TRACK_MODIFICATIONS = True
|
||||
LOG_FILE = 'handler.log'
|
||||
|
||||
# When PURGE_OLD_RECORDS == True then a purging system of old records will
|
||||
# be triggered daily. If this variable is not present, then no purging will be done.
|
||||
# If the purging system is enabled, then the following two variables must be set
|
||||
PURGE_OLD_RECORDS = True
|
||||
# The maximum retention time in days for records stored in the database
|
||||
# which do not match the filters in PURGE_FILTERED_RECORDS.
|
||||
# Use 0 to avoid deletion of all unfiltered records.
|
||||
MAX_DAYS_KEEP_UNFILTERED_RECORDS = 35
|
||||
# See config_example.py for details about PURGE_FILTERED_RECORDS
|
||||
PURGE_FILTERED_RECORDS = {
|
||||
"classification": {
|
||||
"org.clearlinux/hello/world": 1,
|
||||
}
|
||||
}
|
||||
|
||||
# The Telemetry ID (TID) accepted by this `collector` app. The ID should be a
|
||||
# random UUID, generated with (for example) `uuidgen`. The default value
|
||||
# set here is used for records from the Clear Linux OS for Intel
|
||||
# Architecture.
|
||||
TELEMETRY_ID = "6907c830-eed9-4ce9-81ae-76daf8d88f0f"
|
||||
|
||||
|
||||
class Testing(Config):
|
||||
TESTING = True
|
||||
SQLALCHEMY_DATABASE_URI = 'postgresql://postgres:@@db_password@@@localhost/testdb'
|
||||
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
@@ -1,78 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import logging
|
||||
|
||||
|
||||
class Config(object):
|
||||
DEBUG = False
|
||||
TESTING = False
|
||||
LOG_LEVEL = logging.ERROR
|
||||
SQLALCHEMY_DATABASE_URI = 'postgresql://postgres:@@db_password@@@localhost/telemdb'
|
||||
SQLALCHEMY_TRACK_MODIFICATIONS = True
|
||||
LOG_FILE = 'handler.log'
|
||||
|
||||
# When PURGE_OLD_RECORDS == True then a purging system of old records will
|
||||
# be triggered daily. If this variable is not present, then no purging will be done.
|
||||
# If the purging system is enabled, then the following two variables must be set
|
||||
PURGE_OLD_RECORDS = True
|
||||
# The maximum retention time in days for records stored in the database
|
||||
# which do not match the filters in PURGE_FILTERED_RECORDS.
|
||||
# Use 0 to avoid deletion of all unfiltered records.
|
||||
MAX_DAYS_KEEP_UNFILTERED_RECORDS = 35
|
||||
# A dictionary in the following format:
|
||||
# {
|
||||
# "<field_name>": {
|
||||
# "<field_value>": <max_days_to_keep>,
|
||||
# ...
|
||||
# },
|
||||
# ...
|
||||
# }
|
||||
# Currently supported fields to filter:
|
||||
# [
|
||||
# 'severity',
|
||||
# 'classification',
|
||||
# 'machine_id'
|
||||
# ]
|
||||
# Use 0 to avoid deletion of records that matches the filter.
|
||||
# If you do not want to filter records to delete, just set an empty dict '{}'
|
||||
PURGE_FILTERED_RECORDS = {
|
||||
"severity": {
|
||||
1: 5,
|
||||
4: 0
|
||||
},
|
||||
"classification": {
|
||||
"org.clearlinux/mce/*": 0,
|
||||
"org.clearlinux/hello/world": 1,
|
||||
"org.clearlinux/heartbeat/ping": 1,
|
||||
}
|
||||
}
|
||||
|
||||
# The Telemetry ID (TID) accepted by this `collector` app. The ID should be a
|
||||
# random UUID, generated with (for example) `uuidgen`. The default value
|
||||
# set here is used for records from the Clear Linux OS for Intel
|
||||
# Architecture.
|
||||
TELEMETRY_ID = "6907c830-eed9-4ce9-81ae-76daf8d88f0f"
|
||||
|
||||
|
||||
class Testing(Config):
|
||||
TESTING = True
|
||||
SQLALCHEMY_DATABASE_URI = 'postgresql://postgres:@@db_password@@@localhost/testdb'
|
||||
SQLALCHEMY_TRACK_MODIFICATIONS = True
|
||||
PURGE_OLD_RECORDS = True
|
||||
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
@@ -1,61 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
# This configuration file can be used for a local development debug server
|
||||
# at localhost:5000. Overrides the config module.
|
||||
|
||||
import logging
|
||||
|
||||
|
||||
class Config(object):
|
||||
DEBUG = True
|
||||
TESTING = False
|
||||
LOG_LEVEL = logging.ERROR
|
||||
|
||||
# If your telemdb database password is not 'postgres', update this line.
|
||||
SQLALCHEMY_DATABASE_URI = 'postgresql://postgres:postgres@localhost/telemdb'
|
||||
|
||||
SQLALCHEMY_TRACK_MODIFICATIONS = True
|
||||
LOG_FILE = 'handler.log'
|
||||
|
||||
# When PURGE_OLD_RECORDS == True then a purging system of old records will
|
||||
# be triggered daily. If this variable is not present, then no purging will be done.
|
||||
# If the purging system is enabled, then the following two variables must be set
|
||||
PURGE_OLD_RECORDS = True
|
||||
# The maximum retention time in days for records stored in the database
|
||||
# which do not match the filters in PURGE_FILTERED_RECORDS.
|
||||
# Use 0 to avoid deletion of all unfiltered records.
|
||||
MAX_DAYS_KEEP_UNFILTERED_RECORDS = 35
|
||||
# See config_example.py for details about PURGE_FILTERED_RECORDS
|
||||
PURGE_FILTERED_RECORDS = {
|
||||
"classification": {
|
||||
"org.clearlinux/hello/world": 1,
|
||||
}
|
||||
}
|
||||
# Custom Payload transformations
|
||||
# POST_PROCESSING_PARSERS = ["demo"]
|
||||
|
||||
|
||||
class Testing(Config):
|
||||
TESTING = True
|
||||
|
||||
# If your testdb database password is not 'postgres', update this line.
|
||||
SQLALCHEMY_DATABASE_URI = 'postgresql://postgres:postgres@localhost/testdb'
|
||||
|
||||
SQLALCHEMY_TRACK_MODIFICATIONS = True
|
||||
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
@@ -1 +0,0 @@
|
||||
../../shared/crash.py
|
||||
@@ -1,6 +0,0 @@
|
||||
|
||||
|
||||
class PlugablePayloadParserException(Exception):
|
||||
|
||||
def __init__(self, message):
|
||||
self.__str__ = message
|
||||
@@ -1,28 +0,0 @@
|
||||
#
|
||||
# Copyright 2018 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
try:
|
||||
from uwsgidecorators import spool
|
||||
except ImportError:
|
||||
def spool(f):
|
||||
f.spool = f
|
||||
return f
|
||||
|
||||
# Alias for uwsgi spool feature. Using this feature to async data processing from
|
||||
# different probes.
|
||||
# Aliasing this feature will make it easier in the future to move to a different
|
||||
# product if scaling becomes a problem.
|
||||
parser_spooler = spool
|
||||
@@ -1,213 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2018 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import string
|
||||
from .. import app
|
||||
MAX_NUM_RECORDS = '1000'
|
||||
|
||||
REQUIRED_HEADERS_V1 = (
|
||||
'Arch',
|
||||
'Build',
|
||||
'Creation-Timestamp',
|
||||
'Classification',
|
||||
'Host-Type',
|
||||
'Kernel-Version',
|
||||
'Machine-Id',
|
||||
'Severity',
|
||||
'Record-Format-Version',
|
||||
)
|
||||
|
||||
REQUIRED_HEADERS_V2 = REQUIRED_HEADERS_V1 + (
|
||||
'Payload-Format-Version',
|
||||
'System-Name',
|
||||
'X-Telemetry-Tid',
|
||||
)
|
||||
|
||||
REQUIRED_HEADERS_V3 = REQUIRED_HEADERS_V2 + (
|
||||
'Board-Name',
|
||||
'Bios-Version',
|
||||
'Cpu-Model',
|
||||
)
|
||||
|
||||
REQUIRED_HEADERS_V4 = REQUIRED_HEADERS_V3 + (
|
||||
'Event-Id',
|
||||
)
|
||||
|
||||
# see config.py for the meaning of this value
|
||||
TELEMETRY_ID = app.config.get("TELEMETRY_ID", "6907c830-eed9-4ce9-81ae-76daf8d88f0f")
|
||||
|
||||
SEVERITY_VALUES = [1, 2, 3, 4]
|
||||
VALID_RECORD_FORMAT_VERSIONS = [1, 2, 3, 4]
|
||||
POSTGRES_INT_MAX = 2147483647
|
||||
MAXLEN_PRINTABLE = 200
|
||||
|
||||
|
||||
class InvalidUsage(Exception):
|
||||
status_code = 400
|
||||
|
||||
def __init__(self, message, status_code=None, payload=None):
|
||||
Exception.__init__(self)
|
||||
self.message = message
|
||||
if status_code is not None:
|
||||
self.status_code = status_code
|
||||
self.payload = payload
|
||||
app.logger.error("InvalidUsage ({}): {}".format(self.status_code, self.message))
|
||||
|
||||
def to_dict(self):
|
||||
rv = dict(self.payload or ())
|
||||
rv['message'] = self.message
|
||||
return rv
|
||||
|
||||
def __str__(self):
|
||||
return self.message
|
||||
|
||||
|
||||
def validate_headers(headers, required_headers):
|
||||
""" Check for every single required header to be
|
||||
in the request """
|
||||
req_headers = dict(headers)
|
||||
req_headers_keys = req_headers.keys()
|
||||
for header in required_headers:
|
||||
if header not in req_headers_keys:
|
||||
raise InvalidUsage("Record-Format-Version headers are invalid, {} missing".format(header), 400)
|
||||
return True
|
||||
|
||||
|
||||
def is_not_none(v):
|
||||
return v is not None
|
||||
|
||||
|
||||
def is_a_number(n):
|
||||
return str(n).isdigit()
|
||||
|
||||
|
||||
def value_is_printable(a_value):
|
||||
return all([x in string.printable for x in a_value])
|
||||
|
||||
|
||||
def record_format_version_validation(record_format_version):
|
||||
return all([is_not_none(record_format_version), is_a_number(record_format_version)]) and int(record_format_version) in VALID_RECORD_FORMAT_VERSIONS
|
||||
|
||||
|
||||
def record_format_version_headers_validation(record_format_version, headers):
|
||||
# Validate required headers based on Record Version
|
||||
req_headrs = {
|
||||
'1': REQUIRED_HEADERS_V1,
|
||||
'2': REQUIRED_HEADERS_V2,
|
||||
'3': REQUIRED_HEADERS_V3,
|
||||
'4': REQUIRED_HEADERS_V4,
|
||||
}
|
||||
reqs = req_headrs.get(record_format_version, None)
|
||||
if reqs is None:
|
||||
return False
|
||||
return validate_headers(headers, reqs)
|
||||
|
||||
|
||||
def validation_tid_header(tid):
|
||||
return tid == TELEMETRY_ID
|
||||
|
||||
|
||||
def validate_severity(severity):
|
||||
return is_a_number(severity) and int(severity) in SEVERITY_VALUES
|
||||
|
||||
|
||||
def validate_classification(classification):
|
||||
return is_not_none(classification) and len(classification.split('/')) == 3
|
||||
|
||||
|
||||
def validate_machine_id(machine_id):
|
||||
return is_not_none(machine_id) and len(machine_id) <= 32
|
||||
|
||||
|
||||
def validate_timestamp(timestamp):
|
||||
return all([is_not_none(timestamp), is_a_number(timestamp)])
|
||||
|
||||
|
||||
def validate_architecture(arch):
|
||||
return arch in ["armv7l", "armv6l", "aarch64", "amd64", "sparc64", "ppc64", "i686", "i386", "x86_64", "ppc"]
|
||||
|
||||
|
||||
def validate_host_type(host_type):
|
||||
return is_not_none(host_type) and len(host_type) < 250
|
||||
|
||||
|
||||
def validate_kernel_version(kernel_version):
|
||||
""" makes sure that the kernel version string has at least 2 numbers
|
||||
version, major and minor revision
|
||||
"""
|
||||
try:
|
||||
version, major_revision, _ = str(kernel_version).split('.', maxsplit=2)
|
||||
return all([is_a_number(x) for x in [version, major_revision]])
|
||||
except ValueError as e:
|
||||
print(e)
|
||||
return False
|
||||
|
||||
|
||||
def validate_payload_format_version(payload_format_version):
|
||||
return int(payload_format_version) < POSTGRES_INT_MAX
|
||||
|
||||
|
||||
def validate_x_header(header_value):
|
||||
return is_not_none(header_value) and len(header_value) < MAXLEN_PRINTABLE and value_is_printable(header_value)
|
||||
|
||||
|
||||
def validate_created(created):
|
||||
return is_a_number(created)
|
||||
|
||||
|
||||
def validate_record_limit(limit):
|
||||
return is_a_number(limit) and limit <= MAX_NUM_RECORDS
|
||||
|
||||
|
||||
def validate_event_id(header_value):
|
||||
return len(header_value) == 32 and len([v for v in header_value if v in "0123456789abcdef"]) == 32
|
||||
|
||||
|
||||
def validate_header(name, value, expected=None):
|
||||
if expected is None:
|
||||
return {
|
||||
'tid_header': validation_tid_header,
|
||||
'record_format_version': record_format_version_validation,
|
||||
'payload_format_version': validate_payload_format_version,
|
||||
'severity': validate_severity,
|
||||
'classification': validate_classification,
|
||||
'machine_id': validate_machine_id,
|
||||
'timestamp': validate_timestamp,
|
||||
'architecture': validate_architecture,
|
||||
'host_type': validate_host_type,
|
||||
'kernel_version': validate_kernel_version,
|
||||
'board_name': validate_x_header,
|
||||
'cpu_model': validate_x_header,
|
||||
'bios_version': validate_x_header,
|
||||
'build': validate_x_header,
|
||||
'event_id': validate_event_id,
|
||||
}.get(name, lambda x: False)(value)
|
||||
else:
|
||||
return value == expected
|
||||
|
||||
|
||||
def validate_query(name, value):
|
||||
return {
|
||||
'id': is_a_number,
|
||||
'ts_capture': is_a_number,
|
||||
'severity': validate_severity,
|
||||
'classification': validate_classification,
|
||||
'build': validate_x_header,
|
||||
'limit': validate_record_limit,
|
||||
'machine_id': validate_machine_id,
|
||||
'created_in_days': validate_created,
|
||||
'created_in_sec': validate_created,
|
||||
}.get(name, lambda x: False)(value)
|
||||
@@ -1,36 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
from flask import request, current_app, app
|
||||
import logging
|
||||
|
||||
|
||||
@app.before_request
|
||||
def before_request():
|
||||
headers = request.headers
|
||||
payload = request.data
|
||||
print('Before request')
|
||||
current_app.logger.info('\t'.join([
|
||||
datetime.datetime.today().ctime(),
|
||||
request.remote_addr,
|
||||
request.method,
|
||||
request.url,
|
||||
str(request.data),
|
||||
', '.join([': '.join(x) for x in request.headers])])
|
||||
)
|
||||
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
@@ -1 +0,0 @@
|
||||
../../shared/model.py
|
||||
@@ -1,68 +0,0 @@
|
||||
## Plugable payload parsers
|
||||
|
||||
### Overview
|
||||
|
||||
A plugable parser is a python module that encapsulates a data transformation.
|
||||
This transformation is applied to telemetry message payload if the classification
|
||||
matches one of the classifications that the parser defines during plugin
|
||||
registration.
|
||||
|
||||
Plugin registration occurs during collector initialization, to enable a plugin
|
||||
the plugin needs to be listed in the array ```POST_PROCESSING_PARSERS``` in
|
||||
collector configuration.
|
||||
|
||||
For example to enable the ```demo``` parser plugin that comes with the telemetry
|
||||
backend we need to add a line like the following to collector configuration:
|
||||
|
||||
```python
|
||||
|
||||
class Config(object):
|
||||
|
||||
# ... some configuration values
|
||||
|
||||
POST_PROCESSING_PARSERS = ["demo"]
|
||||
|
||||
# ... more configuration values
|
||||
|
||||
```
|
||||
|
||||
|
||||
### Plugable parser `installation`
|
||||
|
||||
The plugable parser module needs to be located under parsers directory, see
|
||||
the following plugin parser source tree:
|
||||
|
||||
```bash
|
||||
<telemetry-root>/collector/collector/
|
||||
parsers/
|
||||
<parser_name_dir>/
|
||||
__init__.py
|
||||
main.py
|
||||
```
|
||||
|
||||
* ```<parser_name_dir>``` is the parser module name and should live under
|
||||
collector/parsers
|
||||
* ```main.py``` is the entry point for plugin registration.
|
||||
|
||||
The parser entry point `must have` the following members:
|
||||
|
||||
* An array called ```CLASSIFICATIONS``` with a list of message classifications. Any message
|
||||
that matches one of these classifications will be parsed by this plugin.
|
||||
|
||||
* A function signature ```parse_payload(kwargs)```
|
||||
which is the transformation that will be applied to the payload.
|
||||
|
||||
### Other considerations
|
||||
|
||||
Plugable parsers are applied asynchronously and no major consideration is required for
|
||||
most use cases. When in doubt keep in mind the following:
|
||||
|
||||
* Payload transformations are applied after the message record (including payload)
|
||||
is safely stored.
|
||||
* It is a correct assumption to think that the entire record can
|
||||
be queried from storage.
|
||||
* It is not possible to register multiple plugins for the same classification,
|
||||
the correct way to go about doing this is to encapsulate the multiple
|
||||
transformations in one plugin. This is by design, remember that transformations
|
||||
are applied asynchronously therefore a given plugin execution order is not
|
||||
guaranteed.
|
||||
@@ -1,35 +0,0 @@
|
||||
#
|
||||
# Copyright 2018 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
|
||||
import time
|
||||
import base64
|
||||
from collector.lib.parser import parser_spooler
|
||||
|
||||
CLASSIFICATIONS = ['org.clearlinux/telemetry/b64payload']
|
||||
|
||||
|
||||
@parser_spooler
|
||||
def parse_payload(**kwargs):
|
||||
"""
|
||||
:param kwargs: classification=<value>,
|
||||
record_id=<value>,
|
||||
payload=<value>
|
||||
:return: None
|
||||
"""
|
||||
print("Processing data")
|
||||
print(base64.b64decode(kwargs.get('payload')))
|
||||
print("Data processed")
|
||||
@@ -1,26 +0,0 @@
|
||||
#
|
||||
# Copyright 2018 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
from collector.models import db
|
||||
|
||||
class Message(db.Model):
|
||||
__tablename__ = 'demo_message'
|
||||
record_id = db.Column(db.Integer, db.ForeignKey('records.id'))
|
||||
payload = buildstamp = db.Column(db.String, default='')
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
self.record_id = kwargs.get('record_id', None)
|
||||
self.payload = kwargs.get('payload', None)
|
||||
@@ -1,38 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
from .model import Record
|
||||
from . import app
|
||||
|
||||
try:
|
||||
import uwsgi
|
||||
from uwsgidecorators import cron
|
||||
|
||||
PURGE_OLD_RECORDS = app.config.get("PURGE_OLD_RECORDS", True)
|
||||
|
||||
# Runs cron job at 4:30 every day
|
||||
@cron(30, 4, -1, -1, -1, target='spooler')
|
||||
def purge_task(signum):
|
||||
if PURGE_OLD_RECORDS:
|
||||
app.logger.info("Running cron job for purging records")
|
||||
with app.app_context():
|
||||
Record.delete_records()
|
||||
|
||||
except ImportError:
|
||||
app.logger.info("Import error for uwsgi")
|
||||
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
@@ -1,346 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import re
|
||||
import time
|
||||
import datetime
|
||||
import importlib
|
||||
from flask import request
|
||||
from flask import jsonify
|
||||
from flask import redirect
|
||||
from .lib.exceptions import PlugablePayloadParserException
|
||||
from .lib.validation import (
|
||||
validate_header,
|
||||
validate_query,
|
||||
MAX_NUM_RECORDS,
|
||||
record_format_version_headers_validation,
|
||||
InvalidUsage)
|
||||
from .model import (
|
||||
Classification,
|
||||
Build)
|
||||
from .crash import (
|
||||
process_guilties,
|
||||
is_crash_classification)
|
||||
from .purge import *
|
||||
|
||||
tm_version_regex = re.compile("^[0-9]+\.[0-9]+$")
|
||||
client_id_regex = re.compile(
|
||||
"^[0-9]+[ \t]+[-/.:_+*a-zA-Z0-9]+[ \t]+[-/.:_+*a-zA-Z0-9]+$")
|
||||
ts_v3_regex = re.compile("^[0-9]+$")
|
||||
# FIXME: configurable limits
|
||||
max_payload_len_inline = 30 * 1024 # 30k
|
||||
max_payload_len = 300 * 1024 # 300k
|
||||
MAX_INTERVAL_SEC = 24 * 60 * 60 * 30 # 30 days in seconds
|
||||
|
||||
|
||||
# This variable is loaded during app initialization from
|
||||
# values on config.POST_PROCESSING_PARSERS
|
||||
POST_PROCESSING_PARSERS = {}
|
||||
|
||||
|
||||
@app.errorhandler(InvalidUsage)
|
||||
def handle_invalid_usage(error):
|
||||
response = jsonify(error.to_dict())
|
||||
response.status_code = error.status_code
|
||||
return response
|
||||
|
||||
|
||||
@app.before_request
|
||||
def before_request():
|
||||
headers = request.headers
|
||||
payload = request.data
|
||||
app.logger.info('\t'.join([
|
||||
datetime.datetime.today().ctime(),
|
||||
request.method,
|
||||
request.url,
|
||||
str(request.data),
|
||||
', '.join([': '.join(x) for x in request.headers])])
|
||||
)
|
||||
|
||||
|
||||
def clean_build_n_value(build):
|
||||
# It is common to see the build numbers quoted in os-release. We don't
|
||||
# need the quotes, so strip them from the semantic value.
|
||||
_build = build.replace('"', '').replace("'", "")
|
||||
return _build
|
||||
|
||||
|
||||
def validate_header_value(header_value, record_name, err_msg):
|
||||
try:
|
||||
if validate_header(record_name, header_value) is True:
|
||||
return header_value
|
||||
except Exception as e:
|
||||
err_msg = "Error parsing {}, {}".format(record_name, e)
|
||||
raise InvalidUsage(err_msg, 400)
|
||||
|
||||
|
||||
def validate_record_v3_headers(board_name, cpu_model, bios_version):
|
||||
validate_header_value(board_name, "board_name", "board name is invalid")
|
||||
validate_header_value(cpu_model, "cpu_model", "cpu model is invalid")
|
||||
validate_header_value(bios_version, "bios_version", "BIOS version is invalid")
|
||||
|
||||
|
||||
def collector_post_handler():
|
||||
|
||||
# The collector only accepts records with the configured TID value.
|
||||
# Make sure the TID in the collector config.py matches the TID
|
||||
# configured for telemetrics-client on the systems from which this
|
||||
# collector receives records.
|
||||
tid_header = request.headers.get("X-Telemetry-TID")
|
||||
validate_header_value(tid_header, "tid_header", "Telemetry ID mismatch")
|
||||
|
||||
record_format_version = request.headers.get("Record-Format-Version")
|
||||
validate_header_value(record_format_version, "record_format_version", "Record-Format-Version is invalid")
|
||||
|
||||
record_format_version_headers_validation(record_format_version, request.headers)
|
||||
|
||||
severity = request.headers.get("Severity")
|
||||
validate_header_value(severity, "severity", "severity value is out of range")
|
||||
|
||||
classification = request.headers.get("Classification")
|
||||
validate_header_value(classification, "classification", "Classification value is invalid")
|
||||
|
||||
machine_id = request.headers.get("Machine-Id")
|
||||
validate_header_value(machine_id, "machine_id", "Machine id value is invalid")
|
||||
|
||||
timestamp = request.headers.get("Creation-Timestamp")
|
||||
validate_header_value(timestamp, "timestamp", "timestamp is invalid")
|
||||
|
||||
ts_capture = int(timestamp)
|
||||
ts_reception = time.time()
|
||||
|
||||
architecture = request.headers.get("Arch")
|
||||
validate_header_value(architecture, "architecture", "architecture is invalid")
|
||||
|
||||
host_type = request.headers.get("Host-Type")
|
||||
validate_header_value(host_type, "host_type", "host type is invalid")
|
||||
|
||||
kernel_version = request.headers.get("Kernel-Version")
|
||||
validate_header_value(kernel_version, "kernel_version", "kernel version is invalid")
|
||||
|
||||
# Record V3 format headers
|
||||
board_name = "N/A"
|
||||
cpu_model = "N/A"
|
||||
bios_version = "N/A"
|
||||
if record_format_version >= '3':
|
||||
board_name = request.headers.get("Board-Name")
|
||||
cpu_model = request.headers.get("Cpu-Model")
|
||||
bios_version = request.headers.get("Bios-Version")
|
||||
validate_record_v3_headers(board_name, cpu_model, bios_version)
|
||||
|
||||
# Record V4 format headers
|
||||
event_id = "N/A"
|
||||
if record_format_version >= '4':
|
||||
event_id = request.headers.get("Event-Id")
|
||||
validate_header_value(event_id, "event_id", "Event id is invalid")
|
||||
|
||||
os_name = request.headers.get('System-Name')
|
||||
os_name = os_name.replace('"', '').replace("'", "")
|
||||
build = request.headers.get('Build')
|
||||
build = clean_build_n_value(build)
|
||||
# The build number is stored as a string in the database, but if this
|
||||
# record is from a Clear Linux OS system, only accept an integer.
|
||||
# Otherwise, loosen the restriction to the characters listed for
|
||||
# VERSION_ID in os-release(5) in addition to the capital letters A-Z.
|
||||
if os_name == 'clear-linux-os':
|
||||
build_regex = re.compile(r"^[0-9]+$")
|
||||
if not build_regex.match(build):
|
||||
raise InvalidUsage("Clear Linux OS build version has invalid characters")
|
||||
else:
|
||||
build_regex = re.compile(r"^[-_a-zA-Z0-9.]+$")
|
||||
if not build_regex.match(build):
|
||||
raise InvalidUsage("Build version has invalid characters")
|
||||
|
||||
payload_format_version = request.headers.get("Payload-Format-Version")
|
||||
validate_header_value(payload_format_version, "payload_format_version",
|
||||
"Payload format version outside of range supported")
|
||||
|
||||
external = request.headers.get('X-CLR-External')
|
||||
if external and external == "true":
|
||||
external = True
|
||||
else:
|
||||
external = False
|
||||
|
||||
try:
|
||||
# prefer UTF-8, if possible
|
||||
payload = request.data.decode('utf-8')
|
||||
except UnicodeError:
|
||||
# fallback to Latin-1, since it accepts all byte values
|
||||
payload = request.data.decode('latin-1')
|
||||
|
||||
db_class = Classification.query.filter_by(classification=classification).first()
|
||||
if db_class is None:
|
||||
db_class = Classification(classification)
|
||||
|
||||
db_build = Build.query.filter_by(build=build).first()
|
||||
if db_build is None:
|
||||
db_build = Build(build)
|
||||
|
||||
db_rec = Record.create(machine_id, host_type, severity, db_class, db_build, architecture, kernel_version,
|
||||
record_format_version, ts_capture, ts_reception, payload_format_version, os_name,
|
||||
board_name, bios_version, cpu_model, event_id, external, payload)
|
||||
|
||||
# TODO: This should become a plugable parser
|
||||
if is_crash_classification(classification):
|
||||
# must pass args as bytes to uwsgi under Python 3
|
||||
process_guilties(klass=classification.encode(), id=str(db_rec.id).encode())
|
||||
|
||||
if classification in POST_PROCESSING_PARSERS.keys():
|
||||
POST_PROCESSING_PARSERS[classification](classification=classification.encode(),
|
||||
record_id=str(db_rec.id).encode(),
|
||||
payload=payload.encode())
|
||||
|
||||
resp = jsonify(db_rec.to_dict())
|
||||
resp.status_code = 201
|
||||
return resp
|
||||
|
||||
|
||||
def validate_query_value(query_value, query_name, err_msg):
|
||||
try:
|
||||
if validate_query(query_name, query_value) is True:
|
||||
return query_value
|
||||
except Exception as e:
|
||||
err_msg = "Error parsing {}, {}".format(query_name, e)
|
||||
raise InvalidUsage(err_msg, 400)
|
||||
|
||||
|
||||
def get_records_api_handler():
|
||||
|
||||
# Validate query parameters correctness
|
||||
severity = request.args.get("severity", None)
|
||||
if severity is not None:
|
||||
validate_query_value(severity, "severity", "Severity should be a numeric value")
|
||||
|
||||
classification = request.args.get('classification', None)
|
||||
if classification is not None:
|
||||
validate_query_value(classification, "classification", "Classification value is invalid")
|
||||
|
||||
build = request.args.get('build', None)
|
||||
if build is not None:
|
||||
build = clean_build_n_value(build)
|
||||
validate_query_value(build, "build", "Build value is invalid")
|
||||
|
||||
machine_id = request.args.get('machine_id', None)
|
||||
if machine_id is not None:
|
||||
validate_query_value(machine_id, "machine_id", "Machine id value is invalid")
|
||||
|
||||
created_in_days = request.args.get('created_in_days', None)
|
||||
if created_in_days is not None:
|
||||
validate_query_value(created_in_days, "created_in_days", "Created (in days) value is invalid")
|
||||
|
||||
created_in_sec = request.args.get('created_in_sec', None)
|
||||
if created_in_sec is not None:
|
||||
validate_query_value(created_in_sec, "created_in_sec", "Created (in seconds) value is invalid")
|
||||
|
||||
from_id = request.args.get('from_id', None)
|
||||
if from_id is not None:
|
||||
validate_query_value(from_id, "id", "Provided record id value is invalid")
|
||||
|
||||
ts_capture = request.args.get('ts_capture', None)
|
||||
if ts_capture is not None:
|
||||
validate_query_value(ts_capture, "ts_capture", "Time stamp from record capture")
|
||||
|
||||
limit = request.args.get('limit', MAX_NUM_RECORDS)
|
||||
if limit != MAX_NUM_RECORDS:
|
||||
validate_query_value(limit, "limit", "Record limit value is invalid")
|
||||
|
||||
# Transform days interval to seconds
|
||||
if created_in_days is not None:
|
||||
created_in_days = int(created_in_days)
|
||||
interval_sec = 24 * 60 * 60 * created_in_days
|
||||
elif created_in_sec is not None:
|
||||
interval_sec = int(created_in_sec)
|
||||
else:
|
||||
interval_sec = MAX_INTERVAL_SEC
|
||||
|
||||
if interval_sec is not None and interval_sec > MAX_INTERVAL_SEC:
|
||||
interval_sec = MAX_INTERVAL_SEC
|
||||
|
||||
records = Record.query_records(build, classification, severity, machine_id, limit, interval_sec,
|
||||
from_id=from_id, ts_capture=ts_capture)
|
||||
record_list = [Record.to_dict(rec) for rec in records]
|
||||
|
||||
return jsonify(records=record_list)
|
||||
|
||||
# ########## Routes ###########
|
||||
|
||||
|
||||
@app.route("/", methods=['GET', 'POST'])
|
||||
@app.route("/v2/collector", methods=['GET', 'POST'])
|
||||
def handler():
|
||||
if request.method == 'POST':
|
||||
return collector_post_handler()
|
||||
else:
|
||||
return redirect("/telemetryui", code=302)
|
||||
|
||||
|
||||
@app.route("/api/records", methods=['GET'])
|
||||
def records_api_handler():
|
||||
"""
|
||||
query filters for simple query:
|
||||
classification
|
||||
severity
|
||||
build
|
||||
machine_id
|
||||
created_in_days - records created after given days
|
||||
created_in_sec - records created after given seconds
|
||||
|
||||
TODO: Advanced query with pagination and following parameters?
|
||||
client_created_after - timestamp
|
||||
client_created_before - timestamp
|
||||
server_created_after - timestamp
|
||||
server_created_before - timestamp
|
||||
|
||||
"""
|
||||
return get_records_api_handler()
|
||||
|
||||
|
||||
def verify_parser_module(parser_module):
|
||||
|
||||
if getattr(parser_module, 'CLASSIFICATIONS', None) is None:
|
||||
raise PlugablePayloadParserException('Parser {} should have a CLASSIFICATIONS field')
|
||||
|
||||
if getattr(parser_module, 'parse_payload', None) is None:
|
||||
raise PlugablePayloadParserException('Parser {} should have a parse_payload method')
|
||||
|
||||
|
||||
def load_parser(parser_name):
|
||||
global POST_PROCESSING_PARSERS
|
||||
|
||||
try:
|
||||
parser_module = importlib.import_module("collector.parsers.{}.main".format(parser_name))
|
||||
verify_parser_module(parser_module)
|
||||
for parser_classification in parser_module.CLASSIFICATIONS:
|
||||
if parser_classification in POST_PROCESSING_PARSERS.keys():
|
||||
raise PlugablePayloadParserException("Parser plugin for class"
|
||||
" {} is already registered".format(parser_classification))
|
||||
POST_PROCESSING_PARSERS[parser_classification] = parser_module.parse_payload
|
||||
app.logger.info(" * Parser: {} registered for class: {}".format(parser_name, parser_classification))
|
||||
except PlugablePayloadParserException as e:
|
||||
print(e.str())
|
||||
except ImportError as ie:
|
||||
print(ie)
|
||||
|
||||
|
||||
def load_parsers():
|
||||
pp_parsers = app.config.get('POST_PROCESSING_PARSERS', [])
|
||||
for pp_parser in pp_parsers:
|
||||
load_parser(pp_parser)
|
||||
|
||||
|
||||
load_parsers()
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
@@ -1,48 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import json
|
||||
import unittest
|
||||
from collector.tests.testcase import (
|
||||
RecordTestCases,
|
||||
get_record)
|
||||
|
||||
|
||||
class TestHandler(RecordTestCases):
|
||||
|
||||
def test_query_report(self):
|
||||
rec = get_record()
|
||||
response = self.client.post('/', headers=rec, data='test')
|
||||
self.assertTrue(response.status_code == 201)
|
||||
filters = {
|
||||
'severity': 1,
|
||||
}
|
||||
response = self.client.get('/api/records', query_string=filters)
|
||||
resp_obj = json.loads(response.data.decode('utf-8'))
|
||||
self.assertEqual(len(resp_obj['records']), 1)
|
||||
filters1 = {
|
||||
'build': '17780',
|
||||
}
|
||||
response = self.client.get('/api/records', query_string=filters1)
|
||||
resp_obj = json.loads(response.data.decode('utf-8'))
|
||||
self.assertEqual(len(resp_obj['records']), 0)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
@@ -1,173 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import unittest
|
||||
import json
|
||||
from collector.tests.testcase import (
|
||||
RecordTestCases,
|
||||
classification,
|
||||
severity,
|
||||
kernel_version,
|
||||
record_version,
|
||||
machine_id,
|
||||
host_type,
|
||||
arch,
|
||||
build,
|
||||
timestamp,
|
||||
tid,
|
||||
board_name,
|
||||
cpu_model,
|
||||
bios_version,
|
||||
system_name,
|
||||
payload_version,
|
||||
event_id,
|
||||
get_record_v1,
|
||||
get_record_v2,
|
||||
get_record_v3,
|
||||
get_record_v4,)
|
||||
|
||||
|
||||
class TestHandlerRecordV2(RecordTestCases):
|
||||
"""
|
||||
Tests record creation for record version 2 and headers validation
|
||||
"""
|
||||
@staticmethod
|
||||
def get_version_records():
|
||||
return get_record_v2()
|
||||
|
||||
def test_record_created(self):
|
||||
headers = get_record_v2()
|
||||
data = "hello"
|
||||
response = self.client.post('/', headers=headers, data=data)
|
||||
self.assertTrue(response.status_code == 201, response.data.decode('utf-8'))
|
||||
json_resp = json.loads(response.data.decode('utf-8'))
|
||||
self.assertTrue(json_resp['classification'] == headers[classification])
|
||||
self.assertTrue(str(json_resp['severity']) == str(headers[severity]))
|
||||
self.assertTrue(json_resp['kernel_version'] == headers[kernel_version])
|
||||
self.assertTrue(str(json_resp['record_format_version']) == str(headers[record_version]))
|
||||
self.assertTrue(json_resp['machine_id'] == headers[machine_id])
|
||||
self.assertTrue(json_resp['machine_type'] == headers[host_type])
|
||||
self.assertTrue(json_resp['arch'] == headers[arch])
|
||||
self.assertTrue(json_resp['build'] == headers[build])
|
||||
self.assertTrue(json_resp['payload'] == data)
|
||||
|
||||
def missing_header(self, header, header_name):
|
||||
headers = get_record_v2()
|
||||
del headers[header_name]
|
||||
response = self.client.post('/', headers=headers, data='test')
|
||||
self.assertTrue(response.status_code == 400, response.data.decode('utf-8'))
|
||||
|
||||
def test_post_fail_missing_classifiction(self):
|
||||
self.missing_header('Classification', severity)
|
||||
|
||||
def test_post_fail_missing_severity(self):
|
||||
self.missing_header('Severity', severity)
|
||||
|
||||
def test_post_fail_missing_kernel_version(self):
|
||||
self.missing_header('Kernel-Version', kernel_version)
|
||||
|
||||
def test_post_fail_missing_host_type(self):
|
||||
self.missing_header('Host-Type', host_type)
|
||||
|
||||
def test_post_fail_missing_machine_id(self):
|
||||
self.missing_header('Machine-Id', machine_id)
|
||||
|
||||
def test_post_fail_missing_arch(self):
|
||||
self.missing_header('Arch', arch)
|
||||
|
||||
def test_post_fail_missing_build(self):
|
||||
self.missing_header('Build', build)
|
||||
|
||||
def test_post_fail_missing_record_version(self):
|
||||
self.missing_header('Record-Format-Version', record_version)
|
||||
|
||||
|
||||
class TestHandlerRecordTransitionV2toV3(RecordTestCases):
|
||||
"""
|
||||
This test case tests a transition where the client is in record
|
||||
version 2 though is sending v3 headers, make sure the server
|
||||
understands the record as v2 and do not have problems creating it
|
||||
"""
|
||||
def test_record_created(self):
|
||||
headers = get_record_v3()
|
||||
data = "hello"
|
||||
response = self.client.post('/', headers=headers, data=data)
|
||||
self.assertTrue(response.status_code == 201, response.data.decode('utf-8'))
|
||||
json_resp = json.loads(response.data.decode('utf-8'))
|
||||
self.assertTrue(json_resp['classification'] == headers[classification])
|
||||
self.assertTrue(int(json_resp['severity']) == int(headers[severity]))
|
||||
self.assertTrue(json_resp['kernel_version'] == headers[kernel_version])
|
||||
self.assertTrue(int(json_resp['record_format_version']) == int(headers[record_version]))
|
||||
self.assertTrue(json_resp['machine_id'] == headers[machine_id])
|
||||
self.assertTrue(json_resp['machine_type'] == headers[host_type])
|
||||
self.assertTrue(json_resp['arch'] == headers[arch])
|
||||
self.assertTrue(json_resp['build'] == headers[build])
|
||||
|
||||
|
||||
class TestHandlerRecordV3(RecordTestCases):
|
||||
"""
|
||||
Test record v3 making sure to validate expected headers for v3
|
||||
if record version is properly set to 3.
|
||||
"""
|
||||
@staticmethod
|
||||
def get_version_records():
|
||||
return get_record_v3()
|
||||
|
||||
def test_post_record_v3_with_headers_v2(self):
|
||||
headers = get_record_v2()
|
||||
headers.update({record_version: 3, })
|
||||
response = self.client.post('/', headers=headers, data='test')
|
||||
self.assertTrue(response.status_code == 400, response.data.decode('utf-8'))
|
||||
|
||||
def test_post_record_v3(self):
|
||||
headers = get_record_v3()
|
||||
response = self.client.post('/', headers=headers, data='test')
|
||||
self.assertTrue(response.status_code == 201, response.data.decode('utf-8'))
|
||||
|
||||
def test_post_missing_cpu_model(self):
|
||||
self.missing_header('Cpu-Model', cpu_model)
|
||||
|
||||
def test_post_missing_board_name(self):
|
||||
self.missing_header('Board-Name', board_name)
|
||||
|
||||
def test_post_missing_bios_version(self):
|
||||
self.missing_header('Bios-Version', bios_version)
|
||||
|
||||
|
||||
class TestHandlerRecordV4(RecordTestCases):
|
||||
"""
|
||||
Test record v4
|
||||
"""
|
||||
@staticmethod
|
||||
def get_version_records():
|
||||
return get_record_v4()
|
||||
|
||||
def test_post_record_v4(self):
|
||||
headers = get_record_v4()
|
||||
response = self.client.post('/', headers=headers, data='test')
|
||||
self.assertTrue(response.status_code == 201, response.data.decode('utf-8'))
|
||||
|
||||
def test_post_missing_eid(self):
|
||||
self.missing_header('Event-Id', event_id)
|
||||
|
||||
|
||||
if __name__ == '__main__' and __package__ is None:
|
||||
from os import sys, path
|
||||
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
|
||||
unittest.main()
|
||||
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
@@ -1,53 +0,0 @@
|
||||
#
|
||||
# Copyright 2018 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import base64
|
||||
import unittest
|
||||
from collector import app
|
||||
from collector.parsers.demo import main
|
||||
from collector.report_handler import load_parsers
|
||||
from collector.tests.testcase import (
|
||||
RecordTestCases,
|
||||
get_record_v3,)
|
||||
|
||||
|
||||
class TestCasesParserPlugin(RecordTestCases):
|
||||
"""
|
||||
Load parsers programmatically for test
|
||||
"""
|
||||
def setUp(self):
|
||||
RecordTestCases.setUp(self)
|
||||
app.config['POST_PROCESSING_PARSERS'] = ["demo"]
|
||||
load_parsers()
|
||||
|
||||
|
||||
class TestParserPlugin(TestCasesParserPlugin):
|
||||
"""
|
||||
Simple test to make sure that plugins parsers are working
|
||||
"""
|
||||
def test_record_created(self):
|
||||
headers = get_record_v3()
|
||||
data = b'Hello World'
|
||||
_data = base64.b64encode(data)
|
||||
headers['classification'] = main.CLASSIFICATIONS
|
||||
response = self.client.post('/', headers=headers, data=_data)
|
||||
self.assertTrue(response.status_code == 201, response.data.decode('utf-8'))
|
||||
|
||||
|
||||
if __name__ == '__main__' and __package__ is None:
|
||||
from os import sys, path
|
||||
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
|
||||
unittest.main()
|
||||
@@ -1,115 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import unittest
|
||||
import time
|
||||
from collector import db
|
||||
from flask import current_app
|
||||
from collector.model import (
|
||||
app,
|
||||
Record,
|
||||
Classification,
|
||||
Build)
|
||||
|
||||
from collector.tests.testcase import (
|
||||
RecordTestCases,
|
||||
get_record,)
|
||||
|
||||
|
||||
def get_insert_params(days_old, severity, classification):
|
||||
record = get_record()
|
||||
record["classification"] = classification
|
||||
record["severity"] = severity
|
||||
db_class = Classification.query.filter_by(classification=record["classification"]).first()
|
||||
if db_class is None:
|
||||
db_class = Classification(record["classification"])
|
||||
db_build = Build.query.filter_by(build=record["build"]).first()
|
||||
if db_build is None:
|
||||
db_build = Build(record["build"])
|
||||
return [
|
||||
record["machine_id"],
|
||||
record["host_type"],
|
||||
record["severity"],
|
||||
db_class,
|
||||
db_build,
|
||||
record["arch"],
|
||||
record["kernel_version"],
|
||||
record["record_format_version"],
|
||||
int(time.time()-3600*24*days_old),
|
||||
int(time.time()-3600*24*days_old),
|
||||
record["payload_format_version"],
|
||||
record["system_name"],
|
||||
record["board_name"],
|
||||
record["bios_version"],
|
||||
record["cpu_model"],
|
||||
"39cc109a1079df96376693ebc7a0f632",
|
||||
False,
|
||||
"Test"
|
||||
]
|
||||
|
||||
|
||||
class TestPurging(RecordTestCases):
|
||||
""" Generic object for telemetry record tests """
|
||||
|
||||
def setUp(self):
|
||||
app.testing = True
|
||||
app.config.from_object('config_local.Testing')
|
||||
app.config["MAX_DAYS_KEEP_UNFILTERED_RECORDS"] = 5
|
||||
app.config["PURGE_FILTERED_RECORDS"] = {
|
||||
"severity": {
|
||||
1: 1,
|
||||
4: 0
|
||||
},
|
||||
"classification": {
|
||||
"test/keep/one": 0,
|
||||
"test/discard/*": 1,
|
||||
}
|
||||
}
|
||||
app.debug = False
|
||||
self.app_context = app.app_context()
|
||||
self.app_context.push()
|
||||
db.init_app(current_app)
|
||||
db.create_all()
|
||||
self.client = app.test_client()
|
||||
|
||||
def test_purge_delete(self):
|
||||
Record.create(*get_insert_params(2, 1, "test/discard/one"))
|
||||
Record.create(*get_insert_params(2, 2, "test/discard/two"))
|
||||
Record.create(*get_insert_params(2, 2, "test/discard/three"))
|
||||
Record.create(*get_insert_params(2, 4, "test/discard/two"))
|
||||
Record.create(*get_insert_params(2, 4, "test/discard/three"))
|
||||
Record.create(*get_insert_params(6, 2, "test/test/one"))
|
||||
Record.create(*get_insert_params(2, 1, "test/keep/one"))
|
||||
self.assertTrue(Record.query.count() == 7)
|
||||
Record.delete_records()
|
||||
self.assertTrue(Record.query.count() == 0)
|
||||
|
||||
def test_purge_keep(self):
|
||||
Record.create(*get_insert_params(6, 2, "test/keep/one"))
|
||||
Record.create(*get_insert_params(3, 2, "test/test/one"))
|
||||
Record.create(*get_insert_params(6, 4, "test/test/one"))
|
||||
Record.create(*get_insert_params(2, 4, "test/discard/three"))
|
||||
Record.create(*get_insert_params(6, 2, "test/test/one"))
|
||||
Record.create(*get_insert_params(2, 1, "test/keep/one"))
|
||||
self.assertTrue(Record.query.count() == 6)
|
||||
Record.delete_records()
|
||||
self.assertTrue(len(Record.query.all()) == 3)
|
||||
|
||||
|
||||
if __name__ == '__main__' and __package__ is None:
|
||||
from os import sys, path
|
||||
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
|
||||
unittest.main()
|
||||
@@ -1,143 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import unittest
|
||||
from collector import (
|
||||
app,
|
||||
db,
|
||||
report_handler,)
|
||||
from flask import current_app
|
||||
|
||||
classification = 'classification'
|
||||
severity = 'severity'
|
||||
kernel_version = 'kernel_version'
|
||||
record_version = 'record_format_version'
|
||||
machine_id = 'machine_id'
|
||||
host_type = 'host_type'
|
||||
arch = 'arch'
|
||||
build = 'build'
|
||||
timestamp = 'creation_timestamp'
|
||||
tid = 'X-Telemetry-Tid'
|
||||
board_name = 'Board-Name'
|
||||
cpu_model = 'Cpu-Model'
|
||||
bios_version = 'Bios-Version'
|
||||
system_name = 'System-Name'
|
||||
payload_version = 'Payload-Format-Version'
|
||||
event_id = 'Event-Id'
|
||||
|
||||
|
||||
REQUIRED_HEADERS_V1 = (
|
||||
'Arch',
|
||||
'Build',
|
||||
'Creation-Timestamp',
|
||||
'Classification',
|
||||
'Host-Type',
|
||||
'Kernel-Version',
|
||||
'Machine-Id',
|
||||
'Severity',
|
||||
'Record-Format-Version',
|
||||
)
|
||||
|
||||
|
||||
def get_record_v1():
|
||||
return {
|
||||
arch: 'x86_64',
|
||||
build: '550',
|
||||
timestamp: 1483232401,
|
||||
classification: 'a/b/c',
|
||||
host_type: 'LenovoT20',
|
||||
kernel_version: '3.16.4-123.generic',
|
||||
machine_id: '1234',
|
||||
severity: 2,
|
||||
record_version: 1,
|
||||
}
|
||||
|
||||
|
||||
def get_record_v2():
|
||||
v2 = get_record_v1()
|
||||
v2.update({
|
||||
record_version: 2,
|
||||
tid: '6907c830-eed9-4ce9-81ae-76daf8d88f0f',
|
||||
system_name: 'clear-linux-os',
|
||||
payload_version: 1
|
||||
})
|
||||
return v2
|
||||
|
||||
|
||||
def get_record_v3():
|
||||
v3 = get_record_v2()
|
||||
v3.update({
|
||||
record_version: 3,
|
||||
board_name: 'D54250WYK|Intel Corporation',
|
||||
cpu_model: 'Intel(R) Core(TM) i5-4250U CPU @ 1.30GHz',
|
||||
bios_version: 'WYLPT10H.86A.0041.2015.0720.1108',
|
||||
|
||||
})
|
||||
return v3
|
||||
|
||||
|
||||
def get_record_v4():
|
||||
v4 = get_record_v3()
|
||||
v4.update({
|
||||
record_version: 4,
|
||||
event_id: '39cc109a1079df96376693ebc7a0f632',
|
||||
})
|
||||
return v4
|
||||
|
||||
|
||||
def get_record():
|
||||
return {
|
||||
"X-Telemetry-TID": "6907c830-eed9-4ce9-81ae-76daf8d88f0f",
|
||||
"record_format_version": "2",
|
||||
"severity": "1",
|
||||
"classification": "org.clearlinux/hello/world",
|
||||
"machine_id": "clr-linux-avj01",
|
||||
"creation_timestamp": "1505235249",
|
||||
"arch": "x86_64",
|
||||
"host_type": "blank|blank|blank",
|
||||
"kernel_version": "4.12.5-374.native",
|
||||
"system_name": "clear-linux-os",
|
||||
"build": "17700",
|
||||
"payload_format_version": "1",
|
||||
"board_name": "D54250WYK|Intel Corporation",
|
||||
"cpu_model": "Intel(R) Core(TM) i5-4250U CPU @ 1.30GHz",
|
||||
"bios_version": "WYLPT10H.86A.0041.2015.0720.1108"
|
||||
}
|
||||
|
||||
|
||||
class RecordTestCases(unittest.TestCase):
|
||||
""" Generic object for telemetry record tests """
|
||||
|
||||
def setUp(self):
|
||||
app.testing = True
|
||||
app.config.from_object('collector.config_local.Testing')
|
||||
app.debug = False
|
||||
self.app_context = app.app_context()
|
||||
self.app_context.push()
|
||||
db.init_app(current_app)
|
||||
db.create_all()
|
||||
self.client = app.test_client()
|
||||
|
||||
def tearDown(self):
|
||||
db.session.remove()
|
||||
db.drop_all()
|
||||
self.app_context.pop()
|
||||
|
||||
def missing_header(self, header, header_name):
|
||||
headers = self.get_version_records()
|
||||
del headers[header_name]
|
||||
response = self.client.post('/', headers=headers, data='test')
|
||||
self.assertTrue(response.status_code == 400)
|
||||
@@ -1,45 +0,0 @@
|
||||
#
|
||||
# Copyright 2015-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import unittest
|
||||
from collector.tests.testcase import (
|
||||
RecordTestCases,
|
||||
get_record_v3,
|
||||
kernel_version)
|
||||
|
||||
|
||||
class TestHandlerRecordValidation(RecordTestCases):
|
||||
"""
|
||||
Test record validation
|
||||
"""
|
||||
@staticmethod
|
||||
def get_version_records():
|
||||
return get_record_v3()
|
||||
|
||||
def test_post_kernel_version_validation_1(self):
|
||||
headers = get_record_v3()
|
||||
headers.update({kernel_version: '3.16.generic'})
|
||||
response = self.client.post('/', headers=headers, data='test')
|
||||
self.assertTrue(response.status_code == 201, response.data.decode('utf-8'))
|
||||
|
||||
|
||||
if __name__ == '__main__' and __package__ is None:
|
||||
from os import sys, path
|
||||
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
|
||||
unittest.main()
|
||||
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
@@ -1,50 +0,0 @@
|
||||
[uwsgi]
|
||||
#application's base folder
|
||||
chdir = @@install_path@@collector/
|
||||
|
||||
#python module to import
|
||||
virtualenv = @@install_path@@venv/
|
||||
module = collector:app
|
||||
plugins = python3
|
||||
|
||||
#socket file's location
|
||||
socket = @@install_path@@collector/%n.sock
|
||||
|
||||
#permissions for the socket file
|
||||
chmod-socket = 644
|
||||
|
||||
processes = 4
|
||||
|
||||
threads = 2
|
||||
|
||||
#enable thread support.Need to figure out the optimal no of threads.
|
||||
enable-threads = true
|
||||
|
||||
#location of log files
|
||||
logto = /var/log/uwsgi/%n.log
|
||||
|
||||
#maximum size of log file before rotation (100MB)
|
||||
log-maxsize = 104857600
|
||||
|
||||
#backup log file (created after rotation)
|
||||
log-backupname = /var/log/uwsgi/%n.log.bk
|
||||
|
||||
#if request takes more than this parameter(in sec), request will be dropped
|
||||
harakiri = 60
|
||||
|
||||
#respawn processes after serving 5000 requests
|
||||
max-requests = 5000
|
||||
|
||||
#get verbose logs when a process gets stuck
|
||||
harakiri-verbose = true
|
||||
|
||||
#http://uwsgi-docs.readthedocs.org/en/latest/Tracebacker.html#combining-the-tracebacker-with-harakiri
|
||||
#Traceback is automatically logged during harakiri phase.
|
||||
py-tracebacker=collectorsocket
|
||||
|
||||
#enable stats. Use this for fine-tuning no of processes.
|
||||
#connect uwsgitop to the stats socket as: uwsgitop /tmp/collectorstats.socket
|
||||
stats=/tmp/collectorstats.socket
|
||||
|
||||
#for asynchronous processing (e.g. updating guilty data for crash records)
|
||||
spooler = %(chdir)/uwsgi-spool
|
||||
+6
-26
@@ -23,7 +23,6 @@ DB_PASSWORD=""
|
||||
NGINX_USER=""
|
||||
NGINX_GROUP=""
|
||||
REPO_NAME="telemetrics-backend"
|
||||
COLLECTOR_INI="collector_uwsgi.ini"
|
||||
TELEMETRYUI_INI="telemetryui_uwsgi.ini"
|
||||
SPOOL_DIR="uwsgi-spool"
|
||||
APT_GET_INSTALL="DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::=\"--force-confnew\""
|
||||
@@ -321,7 +320,6 @@ _stage_from_git() {
|
||||
cd $path
|
||||
rm -rf $REPO_NAME
|
||||
git clone -b $branch $REPO_LOCATION
|
||||
rm -f $REPO_NAME/collector/collector/config_local.py
|
||||
rm -f $REPO_NAME/telemetryui/telemetryui/config_local.py
|
||||
tar -czf backend_local.tar.gz $REPO_NAME
|
||||
)
|
||||
@@ -359,7 +357,6 @@ EOF
|
||||
|
||||
_get_db_pass() {
|
||||
local pass
|
||||
local conf=$REMOTE_APP_DIR/collector/collector/config.py
|
||||
|
||||
if [ ! -f $conf ]; then
|
||||
error "missing config.py file; must run the 'install' action first"
|
||||
@@ -372,7 +369,7 @@ _get_db_pass() {
|
||||
exit 1
|
||||
fi
|
||||
|
||||
pass=$(sed -n -e 's|^.*SQLALCHEMY_DATABASE_URI.*postgres://postgres:\(.*\)@localhost/telemdb.*|\1|p' $conf)
|
||||
pass=$(sed -n -e 's|^.*SQLALCHEMY_DATABASE_URI.*postgres://postgres:\(.*\)@localhost/telemetry.*|\1|p' $conf)
|
||||
echo "$pass"
|
||||
}
|
||||
|
||||
@@ -449,7 +446,6 @@ _get_flask_key() {
|
||||
|
||||
_deploy() {
|
||||
local repo="/tmp/$REPO_NAME"
|
||||
local collector_path="$repo/collector"
|
||||
local telemetryui_path="$repo/telemetryui"
|
||||
local shared_path="$repo/shared"
|
||||
local scripts_path="$repo/scripts"
|
||||
@@ -469,8 +465,7 @@ _deploy() {
|
||||
mv /tmp/backend_local.tar.gz /tmp/backend.tar.gz
|
||||
tar -C /tmp/ -xf /tmp/backend.tar.gz
|
||||
|
||||
# Finalize configuration for collector and telemetryui
|
||||
_subst_config "$scripts_path/$COLLECTOR_INI" "$collector_path/"
|
||||
# Finalize configuration for telemetryui
|
||||
_subst_config "$scripts_path/$TELEMETRYUI_INI" "$telemetryui_path/"
|
||||
_subst_config "$scripts_path/uwsgi.conf"
|
||||
_subst_config "$scripts_path/sites_nginx.conf"
|
||||
@@ -479,13 +474,8 @@ _deploy() {
|
||||
_subst_config "$scripts_path/uwsgi.service"
|
||||
|
||||
# Finalize configuration for postgres
|
||||
_subst_config "$collector_path/collector/config.py"
|
||||
_subst_config "$telemetryui_path/telemetryui/config.py"
|
||||
|
||||
# Install collector + spooldir
|
||||
sudo cp -af $collector_path $REMOTE_APP_DIR/
|
||||
sudo mkdir -pv $REMOTE_APP_DIR/collector/$SPOOL_DIR
|
||||
|
||||
# Install telemetryui + spooldir
|
||||
sudo cp -af $telemetryui_path $REMOTE_APP_DIR/
|
||||
sudo mkdir -pv $REMOTE_APP_DIR/telemetryui/$SPOOL_DIR
|
||||
@@ -500,7 +490,6 @@ _deploy() {
|
||||
_config_uwsgi_${DISTRO}
|
||||
|
||||
# Fix up permissions
|
||||
sudo chown -R $NGINX_USER:$NGINX_GROUP $REMOTE_APP_DIR/collector
|
||||
sudo chown -R $NGINX_USER:$NGINX_GROUP $REMOTE_APP_DIR/telemetryui
|
||||
|
||||
# Modify existing nginx config
|
||||
@@ -509,7 +498,6 @@ _deploy() {
|
||||
# Misc uwsgi config
|
||||
sudo mkdir -pv /etc/uwsgi/vassals
|
||||
sudo ln -sf $REMOTE_APP_DIR/telemetryui/telemetryui_uwsgi.ini /etc/uwsgi/vassals/
|
||||
sudo ln -sf $REMOTE_APP_DIR/collector/collector_uwsgi.ini /etc/uwsgi/vassals/
|
||||
sudo systemctl enable uwsgi
|
||||
|
||||
# Cert configuration
|
||||
@@ -542,14 +530,14 @@ _set_db_pass() {
|
||||
}
|
||||
|
||||
_drop_db() {
|
||||
sudo -u postgres dropdb telemdb
|
||||
sudo -u postgres dropdb telemetry
|
||||
}
|
||||
|
||||
_create_db() {
|
||||
local scripts_path="/tmp/$REPO_NAME/scripts"
|
||||
|
||||
# First, detect if the database already exists. If so, nothing to do here.
|
||||
sudo -u postgres psql --list --pset format=unaligned | grep -q '^telemdb|'
|
||||
sudo -u postgres psql --list --pset format=unaligned | grep -q '^telemetry|'
|
||||
if [ $? -eq 0 ]; then
|
||||
return 0
|
||||
fi
|
||||
@@ -559,20 +547,18 @@ _create_db() {
|
||||
export DB_PASSWORD
|
||||
fi
|
||||
|
||||
sudo -u postgres createdb telemdb > /dev/null 2>&1
|
||||
sudo -u postgres createdb telemetry > /dev/null 2>&1
|
||||
if [ $? -eq 0 ]; then
|
||||
sudo -u postgres psql template1 -c "ALTER USER postgres with encrypted password '$DB_PASSWORD';"
|
||||
else
|
||||
error "failed to create telemdb Postgres database"
|
||||
error "failed to create telemetry Postgres database"
|
||||
_drop_db
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
_update_db_pass() {
|
||||
local src_collector_path="/tmp/$REPO_NAME/collector"
|
||||
local src_telemetryui_path="/tmp/$REPO_NAME/telemetryui"
|
||||
local dest_collector_path="$REMOTE_APP_DIR/collector"
|
||||
local dest_telemetryui_path="$REMOTE_APP_DIR/telemetryui"
|
||||
|
||||
if [ -z "$DB_PASSWORD" ]; then
|
||||
@@ -581,15 +567,9 @@ _update_db_pass() {
|
||||
fi
|
||||
|
||||
# Restore the template files from the backups first
|
||||
if [ -f $src_collector_path/config.py.backup ]; then
|
||||
mv $src_collector_path/config.py.backup $src_collector_path/config.py
|
||||
fi
|
||||
if [ -f $src_telemetryui_path/config.py.backup ]; then
|
||||
mv $src_telemetryui_path/config.py.backup $src_telemetryui_path/config.py
|
||||
fi
|
||||
|
||||
_subst_config "$src_collector_path/config.py" "$dest_collector_path/"
|
||||
_subst_config "$src_telemetryui_path/config.py" "$dest_telemetryui_path/"
|
||||
}
|
||||
|
||||
do_install() {
|
||||
|
||||
@@ -12,14 +12,6 @@ server {
|
||||
ssl_protocols TLSv1.2;
|
||||
ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:AES128-GCM-SHA256:ECDHE-ECDSA-AES128-CCM:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-CCM:AES128-CCM:AES128-SHA256:AES256-CCM:AES256-SHA256:DHE-RSA-AES128-CCM:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-CCM:DHE-RSA-AES256-SHA256:DHE-DSS-AES128-GCM-SHA256:DHE-DSS-AES128-SHA256:DHE-DSS-AES256-SHA256";
|
||||
|
||||
location / { try_files $uri @collectorapp; }
|
||||
location /v2/collector { try_files $uri @collectorapp; }
|
||||
location @collectorapp {
|
||||
include uwsgi_params;
|
||||
uwsgi_pass unix:@@install_path@@collector/collector_uwsgi.sock;
|
||||
error_log /var/log/nginx/collector_error.log;
|
||||
}
|
||||
|
||||
location /telemetryui { try_files $uri @uiapp; }
|
||||
location @uiapp {
|
||||
include uwsgi_params;
|
||||
|
||||
+4
-4
@@ -232,12 +232,12 @@ def _process_guilties(args):
|
||||
crashes = Record.get_new_crash_records(classes=get_backtrace_classes(), id=record_id)
|
||||
filters = GuiltyBlacklist.get_guilties()
|
||||
for rec in crashes:
|
||||
if rec.backtrace:
|
||||
new_bt = demangle_backtrace(rec.backtrace)
|
||||
rec.backtrace = new_bt
|
||||
if rec.payload:
|
||||
new_bt = demangle_backtrace(rec.payload)
|
||||
rec.payload = new_bt
|
||||
# TODO: update the rec.payload field as well
|
||||
Record.commit_guilty_changes()
|
||||
g, match = find_guilty(rec.backtrace)
|
||||
g, match = find_guilty(rec.payload)
|
||||
if match:
|
||||
function = g['function']
|
||||
module = g['module']
|
||||
|
||||
+119
-160
@@ -19,7 +19,6 @@ from flask_sqlalchemy import SQLAlchemy
|
||||
from sqlalchemy.sql.expression import cast
|
||||
from sqlalchemy.sql.expression import desc
|
||||
from sqlalchemy.sql.expression import case
|
||||
from sqlalchemy.sql import text
|
||||
from time import time, localtime, strftime, mktime, strptime, gmtime
|
||||
from distutils.version import LooseVersion
|
||||
|
||||
@@ -28,27 +27,6 @@ from . import app
|
||||
db = SQLAlchemy(app)
|
||||
|
||||
|
||||
|
||||
class Classification(db.Model):
|
||||
__tablename__ = 'classification'
|
||||
id = db.Column(db.Integer, primary_key=True)
|
||||
classification = db.Column(db.String)
|
||||
domain = db.Column(db.String)
|
||||
probe = db.Column(db.String)
|
||||
|
||||
def __init__(self, class_str):
|
||||
self.classification = class_str
|
||||
|
||||
|
||||
class Build(db.Model):
|
||||
__tablename__ = 'build'
|
||||
id = db.Column(db.Integer, primary_key=True)
|
||||
build = db.Column(db.String)
|
||||
|
||||
def __init__(self, build_num):
|
||||
self.build = build_num
|
||||
|
||||
|
||||
class Guilty(db.Model):
|
||||
__tablename__ = 'guilty'
|
||||
id = db.Column(db.Integer, primary_key=True)
|
||||
@@ -99,56 +77,46 @@ class Guilty(db.Model):
|
||||
class Record(db.Model):
|
||||
__tablename__ = 'records'
|
||||
id = db.Column(db.Integer, primary_key=True)
|
||||
severity = db.Column(db.Integer)
|
||||
machine = db.Column(db.String, default='')
|
||||
machine_id = db.Column(db.String, default='')
|
||||
architecture = db.Column(db.String)
|
||||
kernel_version = db.Column(db.String, default=0)
|
||||
os_name = db.Column(db.String)
|
||||
record_format_version = db.Column(db.Integer, default=0)
|
||||
payload_format_version = db.Column(db.Integer)
|
||||
payload = db.Column(db.LargeBinary)
|
||||
tsp = db.Column(db.Integer)
|
||||
tsp_server = db.Column(db.Integer)
|
||||
buildstamp = db.Column(db.String, default='')
|
||||
backtrace = db.Column(db.String, default='')
|
||||
guilty = db.Column(db.String, default='')
|
||||
dupe_of = db.Column(db.Integer, default=0)
|
||||
dupecount = db.Column(db.Integer, default=0)
|
||||
dupemaster = db.Column(db.Boolean, default=False)
|
||||
security = db.Column(db.Boolean, default=False)
|
||||
hide = db.Column(db.Boolean, default=False)
|
||||
processed = db.Column(db.Boolean, default=False)
|
||||
icon = db.Column(db.String, default='')
|
||||
classification_id = db.Column(db.Integer, db.ForeignKey('classification.id'))
|
||||
build_id = db.Column(db.Integer, db.ForeignKey('build.id'))
|
||||
guilty_id = db.Column(db.Integer, db.ForeignKey('guilty.id'))
|
||||
architecture = db.Column(db.Text)
|
||||
bios_version = db.Column(db.Text, default='')
|
||||
board_name = db.Column(db.Text, default='')
|
||||
build = db.Column(db.Text, nullable=False)
|
||||
classification = db.Column(db.Text, nullable=False)
|
||||
cpu_model = db.Column(db.Text, default='')
|
||||
event_id = db.Column(db.Text, default='')
|
||||
external = db.Column(db.Boolean, default=False)
|
||||
board_name = db.Column(db.String, default='')
|
||||
bios_version = db.Column(db.String, default='')
|
||||
cpu_model = db.Column(db.String, default='')
|
||||
event_id = db.Column(db.String, default='')
|
||||
host_type = db.Column(db.Text, default='')
|
||||
kernel_version = db.Column(db.Text, default=0)
|
||||
machine_id = db.Column(db.Text, default='')
|
||||
payload_version = db.Column(db.Integer)
|
||||
record_version = db.Column(db.Integer, default=0)
|
||||
severity = db.Column(db.Integer)
|
||||
system_name = db.Column(db.Text)
|
||||
timestamp_client = db.Column(db.Numeric)
|
||||
timestamp_server = db.Column(db.Numeric, nullable=False)
|
||||
payload = db.Column(db.Text, nullable=False)
|
||||
|
||||
classification = db.relationship('Classification', backref=db.backref('records', lazy='dynamic'), lazy='joined')
|
||||
build = db.relationship('Build', backref=db.backref('records', lazy='dynamic'), lazy='joined')
|
||||
processed = db.Column(db.Boolean, default=False)
|
||||
guilty_id = db.Column(db.Integer, db.ForeignKey('guilty.id'))
|
||||
|
||||
guilty = db.Column(db.Text, default='')
|
||||
guilty = db.relationship('Guilty', backref=db.backref('records', lazy='dynamic'), lazy='joined')
|
||||
|
||||
def __init__(self, machine_id, host_type, severity, classification, build, architecture, kernel_version,
|
||||
record_format_version, ts_capture, ts_reception, payload_format_version, os_name,
|
||||
record_version, ts_capture, ts_reception, payload_version, system_name,
|
||||
board_name, bios_version, cpu_model, event_id, external, payload):
|
||||
self.machine_id = machine_id
|
||||
self.machine = host_type
|
||||
self.host_type = host_type
|
||||
self.architecture = architecture
|
||||
self.classification = classification
|
||||
self.backtrace = payload
|
||||
self.build = build
|
||||
self.kernel_version = kernel_version
|
||||
self.record_format_version = record_format_version
|
||||
self.record_version = record_version
|
||||
self.severity = severity
|
||||
self.tsp = ts_capture
|
||||
self.tsp_server = ts_reception
|
||||
self.payload_format_version = payload_format_version
|
||||
self.os_name = os_name
|
||||
self.timestamp_client = ts_capture
|
||||
self.timestamp_server = ts_reception
|
||||
self.payload_version = payload_version
|
||||
self.system_name = system_name
|
||||
self.external = external
|
||||
self.board_name = board_name
|
||||
self.bios_version = bios_version
|
||||
@@ -161,7 +129,7 @@ class Record(db.Model):
|
||||
self.payload = payload.encode('latin-1')
|
||||
|
||||
def __repr__(self):
|
||||
return "<Record(id='{}', class='{}', build='{}', created='{}')>".format(self.id, self.classification, self.build, strftime("%a, %d %b %Y %H:%M:%S", localtime(self.tsp)))
|
||||
return "<Record(id='{}', class='{}', build='{}', created='{}')>".format(self.id, self.classification, self.build, strftime("%a, %d %b %Y %H:%M:%S", localtime(self.timestamp_client)))
|
||||
|
||||
def __str__(self):
|
||||
return str(self.to_dict())
|
||||
@@ -170,16 +138,16 @@ class Record(db.Model):
|
||||
record = {
|
||||
'id': self.id,
|
||||
'machine_id': self.machine_id,
|
||||
'machine_type': self.machine,
|
||||
'machine_type': self.host_type,
|
||||
'arch': self.architecture,
|
||||
'build': self.build.build,
|
||||
'build': self.build,
|
||||
'kernel_version': self.kernel_version,
|
||||
'ts_capture': strftime('%Y-%m-%d %H:%M:%S UTC', gmtime(self.tsp)),
|
||||
'ts_reception': strftime('%Y-%m-%d %H:%M:%S UTC', gmtime(self.tsp_server)),
|
||||
'ts_capture': strftime('%Y-%m-%d %H:%M:%S UTC', gmtime(self.timestamp_client)),
|
||||
'ts_reception': strftime('%Y-%m-%d %H:%M:%S UTC', gmtime(self.timestamp_server)),
|
||||
'severity': self.severity,
|
||||
'classification': self.classification.classification,
|
||||
'record_format_version': self.record_format_version,
|
||||
'payload': self.backtrace,
|
||||
'classification': self.classification,
|
||||
'record_version': self.record_version,
|
||||
'payload': self.payload,
|
||||
'board_name': self.board_name,
|
||||
'bios_version': self.bios_version,
|
||||
'cpu_model': self.cpu_model,
|
||||
@@ -193,12 +161,12 @@ class Record(db.Model):
|
||||
record = [
|
||||
self.id,
|
||||
self.external,
|
||||
self.tsp_server,
|
||||
self.timestamp_server,
|
||||
self.severity,
|
||||
self.classification.classification,
|
||||
self.build.build,
|
||||
self.classification,
|
||||
self.build,
|
||||
self.machine_id,
|
||||
self.backtrace
|
||||
self.payload
|
||||
]
|
||||
return record
|
||||
|
||||
@@ -208,11 +176,11 @@ class Record(db.Model):
|
||||
|
||||
@staticmethod
|
||||
def create(machine_id, host_type, severity, classification, build, architecture, kernel_version,
|
||||
record_format_version, ts_capture, ts_reception, payload_format_version, os_name,
|
||||
record_version, ts_capture, ts_reception, payload_version, system_name,
|
||||
board_name, bios_version, cpu_model, event_id, external, payload):
|
||||
try:
|
||||
record = Record(machine_id, host_type, severity, classification, build, architecture, kernel_version,
|
||||
record_format_version, ts_capture, ts_reception, payload_format_version, os_name,
|
||||
record_version, ts_capture, ts_reception, payload_version, system_name,
|
||||
board_name, bios_version, cpu_model, event_id, external, payload)
|
||||
db.session.add(record)
|
||||
db.session.commit()
|
||||
@@ -226,9 +194,9 @@ class Record(db.Model):
|
||||
interval_sec=None, ts_capture=None, from_id=None):
|
||||
records = Record.query
|
||||
if build is not None:
|
||||
records = records.join(Record.build).filter_by(build=build)
|
||||
records = records.filter_by(build=build)
|
||||
if classification is not None:
|
||||
records = records.join(Record.classification).filter_by(classification=classification)
|
||||
records = records.filter_by(classification=classification)
|
||||
if severity is not None:
|
||||
records = records.filter(Record.severity == severity)
|
||||
if machine_id is not None:
|
||||
@@ -236,15 +204,15 @@ class Record(db.Model):
|
||||
if from_id is not None:
|
||||
records = records.filter(Record.id >= from_id)
|
||||
if ts_capture is not None:
|
||||
records = records.filter(Record.tsp > ts_capture)
|
||||
records = records.filter(Record.timestamp_client > ts_capture)
|
||||
|
||||
if interval_sec is not None:
|
||||
current_time = time()
|
||||
secs_in_past = current_time - interval_sec
|
||||
# Due to time skew on client systems, delayed sends due to
|
||||
# spooling, etc, tsp_server works better as the reference
|
||||
# spooling, etc, timestamp_server works better as the reference
|
||||
# timestamp.
|
||||
records = records.filter(Record.tsp_server > secs_in_past)
|
||||
records = records.filter(Record.timestamp_server > secs_in_past)
|
||||
|
||||
records = records.order_by(Record.id.desc())
|
||||
|
||||
@@ -259,32 +227,32 @@ class Record(db.Model):
|
||||
return record
|
||||
|
||||
@staticmethod
|
||||
def filter_records(build, classification, severity, machine_id=None, os_name=None, limit=None, from_date=None,
|
||||
def filter_records(build, classification, severity, machine_id=None, system_name=None, limit=None, from_date=None,
|
||||
to_date=None, payload=None, not_payload=None, data_source=None):
|
||||
records = Record.query
|
||||
if build is not None:
|
||||
records = records.join(Record.build).filter_by(build=build)
|
||||
records = records.filter_by(build=build)
|
||||
if classification is not None:
|
||||
if isinstance(classification, list):
|
||||
records = records.join(Record.classification).filter(Classification.classification.in_(classification))
|
||||
records = records.filter(Record.classification.in_(classification))
|
||||
else:
|
||||
records = records.join(Record.classification).filter(Classification.classification.like(classification))
|
||||
records = records.filter(Record.classification.like(classification))
|
||||
if severity is not None:
|
||||
records = records.filter(Record.severity == severity)
|
||||
if os_name is not None:
|
||||
records = records.filter(Record.os_name == os_name)
|
||||
if system_name is not None:
|
||||
records = records.filter(Record.system_name == system_name)
|
||||
if machine_id is not None:
|
||||
records = records.filter(Record.machine_id == machine_id)
|
||||
if from_date is not None:
|
||||
from_date = mktime(strptime(from_date, "%Y-%m-%d"))
|
||||
records = records.filter(Record.tsp >= from_date)
|
||||
records = records.filter(Record.timestamp_client >= from_date)
|
||||
if to_date is not None:
|
||||
to_date = mktime(strptime(to_date, "%Y-%m-%d"))
|
||||
records = records.filter(Record.tsp < to_date)
|
||||
records = records.filter(Record.timestamp_client < to_date)
|
||||
if payload is not None:
|
||||
records = records.filter(Record.backtrace.op('~')(payload))
|
||||
records = records.filter(Record.payload.op('~')(payload))
|
||||
if not_payload is not None:
|
||||
records = records.filter(~Record.backtrace.op('~')(not_payload))
|
||||
records = records.filter(~Record.payload.op('~')(not_payload))
|
||||
if data_source is not None:
|
||||
if data_source == "external":
|
||||
records = records.filter(Record.external == True)
|
||||
@@ -309,11 +277,10 @@ class Record(db.Model):
|
||||
age = time() - PURGE_FILTERED_RECORDS[field][name] * 24 * 60 * 60
|
||||
q = db.session.query(Record)
|
||||
if field == 'classification':
|
||||
q = q.join(Record.classification)
|
||||
q = q.filter(Classification.classification.like(name.replace("*", "%")))
|
||||
q = q.filter(Record.classification.like(name.replace("*", "%")))
|
||||
else:
|
||||
q = q.filter(getattr(Record, field) == name)
|
||||
q = q.filter(Record.tsp_server < age)
|
||||
q = q.filter(Record.timestamp_server < age)
|
||||
if q.all():
|
||||
count = db.session.query(Record).filter(Record.id.in_([x.id for x in q.all()])).delete(synchronize_session=False)
|
||||
print("Deleted {} {} records".format(count, name))
|
||||
@@ -324,13 +291,12 @@ class Record(db.Model):
|
||||
q = db.session.query(Record.id)
|
||||
for field in PURGE_FILTERED_RECORDS.keys():
|
||||
if field == 'classification':
|
||||
q = q.join(Record.classification)
|
||||
for classification in PURGE_FILTERED_RECORDS[field].keys():
|
||||
q = q.filter(~Classification.classification.like(classification.replace("*", "%")))
|
||||
q = q.filter(~Record.classification.like(classification.replace("*", "%")))
|
||||
else:
|
||||
for name in PURGE_FILTERED_RECORDS[field].keys():
|
||||
q = q.filter(getattr(Record, field) != name)
|
||||
q = q.filter(Record.tsp_server < unfiltered_age)
|
||||
q = q.filter(Record.timestamp_server < unfiltered_age)
|
||||
if q.all():
|
||||
count = db.session.query(Record).filter(Record.id.in_([x.id for x in q.all()])).delete(synchronize_session=False)
|
||||
print("Deleted {} old records".format(count))
|
||||
@@ -342,22 +308,21 @@ class Record(db.Model):
|
||||
|
||||
@staticmethod
|
||||
def get_recordcnts_by_build():
|
||||
q = db.session.query(Build.build, db.func.count(Record.id)).join(Record.build)
|
||||
q = q.filter(Build.build.op('~')('^[0-9]+$'))
|
||||
q = q.group_by(Build.build).order_by(cast(Build.build, db.Integer)).all()
|
||||
q = db.session.query(Record.build, db.func.count(Record.id))
|
||||
q = q.filter(Record.build.op('~')('^[0-9]+$'))
|
||||
q = q.group_by(Record.build).order_by(cast(Record.build, db.Integer)).all()
|
||||
return q
|
||||
|
||||
@staticmethod
|
||||
def get_builds():
|
||||
q = db.session.query(Build.build)
|
||||
q = q.order_by(Build.build)
|
||||
q = db.session.query(Record.build).distinct()
|
||||
q = q.order_by(Record.build)
|
||||
return sorted(q.all(), key=lambda x: LooseVersion(x[0]), reverse=True)
|
||||
|
||||
@staticmethod
|
||||
def get_recordcnts_by_classification():
|
||||
q = db.session.query(Classification.classification, db.func.count(Record.id).label('total'))
|
||||
q = q.join(Record.classification)
|
||||
q = q.group_by(Classification.classification)
|
||||
q = db.session.query(Record.classification, db.func.count(Record.id).label('total'))
|
||||
q = q.group_by(Record.classification)
|
||||
q = q.order_by(desc('total'))
|
||||
return q.all()
|
||||
|
||||
@@ -368,7 +333,7 @@ class Record(db.Model):
|
||||
|
||||
@staticmethod
|
||||
def get_classifications(with_regex=False):
|
||||
q = db.session.query(Classification.classification)
|
||||
q = db.session.query(Record.classification).distinct()
|
||||
if with_regex:
|
||||
classes = [Record.expand_class(c[0].split('/')) for c in q.all()]
|
||||
return sorted(set(itertools.chain(*classes)))
|
||||
@@ -377,7 +342,7 @@ class Record(db.Model):
|
||||
|
||||
@staticmethod
|
||||
def get_os_map():
|
||||
q = db.session.query(Record.os_name, Build.build).join(Record.build).order_by(Record.os_name).group_by(Record.os_name, Build.build).all()
|
||||
q = db.session.query(Record.system_name, Record.build).order_by(Record.system_name).group_by(Record.system_name, Record.build).all()
|
||||
result = {}
|
||||
for x in q:
|
||||
result.setdefault(x[0], []).append(x[1])
|
||||
@@ -385,8 +350,8 @@ class Record(db.Model):
|
||||
|
||||
@staticmethod
|
||||
def get_recordcnts_by_machine_type():
|
||||
q = db.session.query(Record.machine, db.func.count(Record.id).label('total'))
|
||||
q = q.group_by(Record.machine)
|
||||
q = db.session.query(Record.host_type, db.func.count(Record.id).label('total'))
|
||||
q = q.group_by(Record.host_type)
|
||||
q = q.order_by(desc('total'))
|
||||
return q.all()
|
||||
|
||||
@@ -397,55 +362,53 @@ class Record(db.Model):
|
||||
|
||||
@staticmethod
|
||||
def get_crashcnts_by_class(classes=None):
|
||||
q = db.session.query(Classification.classification, db.func.count(Record.id))
|
||||
q = q.join(Record)
|
||||
q = db.session.query(Record.classification, db.func.count(Record.id))
|
||||
if classes:
|
||||
q = q.filter(Classification.classification.in_(classes))
|
||||
q = q.filter(Record.classification.in_(classes))
|
||||
else:
|
||||
q = q.filter(Classification.classification.like('org.clearlinux/crash/%'))
|
||||
q = q.group_by(Classification.classification)
|
||||
q = q.filter(Record.classification.like('org.clearlinux/crash/%'))
|
||||
q = q.group_by(Record.classification)
|
||||
return q.all()
|
||||
|
||||
@staticmethod
|
||||
def get_crashcnts_by_build(classes=None):
|
||||
q = db.session.query(Build.build, db.func.count(Record.id)).join(Record).join(Classification)
|
||||
q = db.session.query(Record.build, db.func.count(Record.id))
|
||||
if not classes:
|
||||
classes = ['org.clearlinux/crash/clr']
|
||||
q = q.filter(Classification.classification.in_(classes))
|
||||
q = q.filter(Build.build.op('~')('^[0-9]+$'))
|
||||
q = q.group_by(Build.build)
|
||||
q = q.order_by(desc(cast(Build.build, db.Integer)))
|
||||
q = q.filter(Record.classification.in_(classes))
|
||||
q = q.filter(Record.build.op('~')('^[0-9]+$'))
|
||||
q = q.group_by(Record.build)
|
||||
q = q.order_by(desc(cast(Record.build, db.Integer)))
|
||||
q = q.limit(10)
|
||||
return q.all()
|
||||
|
||||
@staticmethod
|
||||
def get_top_crash_guilties(classes=None):
|
||||
q = db.session.query(Guilty.function, Guilty.module, Build.build, db.func.count(Record.id).label('total'), Guilty.id, Guilty.comment)
|
||||
q = db.session.query(Guilty.function, Guilty.module, Record.build, db.func.count(Record.id).label('total'), Guilty.id, Guilty.comment)
|
||||
q = q.join(Record)
|
||||
q = q.join(Build)
|
||||
q = q.join(Classification).filter(Record.classification_id == Classification.id)
|
||||
if not classes:
|
||||
classes = ['org.clearlinux/crash/clr']
|
||||
q = q.filter(Classification.classification.in_(classes))
|
||||
q = q.filter(Build.build.op('~')('^[0-9][0-9]+$'))
|
||||
q = q.filter(Record.classification.in_(classes))
|
||||
q = q.filter(Record.build.op('~')('^[0-9][0-9]+$'))
|
||||
q = q.filter(cast(Record.build, db.Integer) <= 100000)
|
||||
q = q.filter(Guilty.hide == False)
|
||||
q = q.group_by(Guilty.function, Guilty.module, Guilty.comment, Guilty.id, Build.build)
|
||||
q = q.order_by(desc(cast(Build.build, db.Integer)), desc('total'))
|
||||
q = q.group_by(Guilty.function, Guilty.module, Guilty.comment, Guilty.id, Record.build)
|
||||
q = q.order_by(desc(cast(Record.build, db.Integer)), desc('total'))
|
||||
# query for records created in the last week (~ 10 Clear builds)
|
||||
q = q.filter(Build.build.in_(sorted(tuple(set([x[2] for x in q.all()])), key=lambda x: int(x))[-8:]))
|
||||
q = q.filter(Record.build.in_(sorted(tuple(set([x[2] for x in q.all()])), key=lambda x: int(x))[-8:]))
|
||||
interval_sec = 24 * 60 * 60 * 7
|
||||
current_time = time()
|
||||
sec_in_past = current_time - interval_sec
|
||||
q = q.filter(Record.tsp > sec_in_past)
|
||||
q = q.filter(Record.timestamp_client > sec_in_past)
|
||||
return q.all()
|
||||
|
||||
@staticmethod
|
||||
def get_new_crash_records(classes=None, id=None):
|
||||
q = db.session.query(Record).join(Classification)
|
||||
q = db.session.query(Record)
|
||||
if not classes:
|
||||
classes = ['org.clearlinux/crash/clr']
|
||||
q = q.filter(Classification.classification.in_(classes))
|
||||
q = q.filter(Record.os_name == 'clear-linux-os')
|
||||
q = q.filter(Record.classification.in_(classes))
|
||||
q = q.filter(Record.system_name == 'clear-linux-os')
|
||||
q = q.filter(Record.processed == False)
|
||||
if id:
|
||||
q = q.filter(Record.id == id)
|
||||
@@ -481,18 +444,17 @@ class Record(db.Model):
|
||||
|
||||
@staticmethod
|
||||
def get_crash_backtraces(classes=None, guilty_id=None, machine_id=None, build=None, most_recent=None, record_id=None):
|
||||
q = db.session.query(Record.backtrace, Record.id).join(Classification)
|
||||
q = db.session.query(Record.payload, Record.id)
|
||||
# Short circuit if we know the record ID
|
||||
if record_id:
|
||||
q = q.filter(Record.id == record_id)
|
||||
return q.first()
|
||||
if build:
|
||||
q = q.join(Build)
|
||||
q = q.filter(Build.build == build)
|
||||
q = q.filter(Record.build == build)
|
||||
if not classes:
|
||||
classes = ['org.clearlinux/crash/clr']
|
||||
q = q.filter(Classification.classification.in_(classes))
|
||||
q = q.filter(Record.os_name == 'clear-linux-os')
|
||||
q = q.filter(Record.classification.in_(classes))
|
||||
q = q.filter(Record.system_name == 'clear-linux-os')
|
||||
if guilty_id:
|
||||
q = q.filter(Record.guilty_id == guilty_id)
|
||||
if machine_id:
|
||||
@@ -501,16 +463,16 @@ class Record(db.Model):
|
||||
interval_sec = 24 * 60 * 60 * int(most_recent)
|
||||
current_time = time()
|
||||
sec_in_past = current_time - interval_sec
|
||||
q = q.filter(Record.tsp > sec_in_past)
|
||||
q = q.filter(Record.timestamp_client > sec_in_past)
|
||||
return q.all()
|
||||
|
||||
@staticmethod
|
||||
def reset_processed_records(classes=None, id=None):
|
||||
q = db.session.query(Record).join(Classification).join(Build)
|
||||
q = db.session.query(Record)
|
||||
if not classes:
|
||||
classes = ['org.clearlinux/crash/clr']
|
||||
q = q.filter(Classification.classification.in_(classes))
|
||||
q = q.filter(Record.os_name == 'clear-linux-os')
|
||||
q = q.filter(Record.classification.in_(classes))
|
||||
q = q.filter(Record.system_name == 'clear-linux-os')
|
||||
if id:
|
||||
q = q.filter(Record.id == id)
|
||||
records = q.all()
|
||||
@@ -520,45 +482,44 @@ class Record(db.Model):
|
||||
|
||||
@staticmethod
|
||||
def get_machine_ids_for_guilty(id, most_recent=None):
|
||||
q = db.session.query(Build.build, Record.machine_id, db.func.count(Record.id).label('total'), Record.guilty_id)
|
||||
q = q.join(Record)
|
||||
q = db.session.query(Record.build, Record.machine_id, db.func.count(Record.id).label('total'), Record.guilty_id)
|
||||
q = q.filter(Record.guilty_id == id)
|
||||
q = q.filter(Record.os_name == 'clear-linux-os')
|
||||
q = q.filter(Build.build.op('~')('^[0-9][0-9]+$'))
|
||||
q = q.group_by(Build.build, Record.machine_id, Record.guilty_id)
|
||||
q = q.order_by(desc(cast(Build.build, db.Integer)), desc('total'))
|
||||
q = q.filter(Record.system_name == 'clear-linux-os')
|
||||
q = q.filter(Record.build.op('~')('^[0-9][0-9]+$'))
|
||||
q = q.group_by(Record.build, Record.machine_id, Record.guilty_id)
|
||||
q = q.order_by(desc(cast(Record.build, db.Integer)), desc('total'))
|
||||
if most_recent:
|
||||
interval_sec = 24 * 60 * 60 * int(most_recent)
|
||||
current_time = time()
|
||||
sec_in_past = current_time - interval_sec
|
||||
q = q.filter(Record.tsp > sec_in_past)
|
||||
q = q.filter(Record.timestamp_client > sec_in_past)
|
||||
return q.all()
|
||||
|
||||
@staticmethod
|
||||
def get_update_msgs():
|
||||
q = db.session.query(Record.backtrace).join(Classification)
|
||||
q = q.filter(Classification.classification == "org.clearlinux/swupd-client/update")
|
||||
q = db.session.query(Record.payload)
|
||||
q = q.filter(Record.classification == "org.clearlinux/swupd-client/update")
|
||||
|
||||
sec_2_weeks = 24 * 60 * 60 * 7
|
||||
current_time = time()
|
||||
time_2_weeks_ago = current_time - sec_2_weeks
|
||||
|
||||
# query for records created in that last 2 weeks
|
||||
q = q.filter(Record.tsp > time_2_weeks_ago)
|
||||
q = q.filter(Record.timestamp_client > time_2_weeks_ago)
|
||||
return q
|
||||
|
||||
@staticmethod
|
||||
def get_swupd_msgs(most_recent=None):
|
||||
q = db.session.query(Record.tsp, Record.machine_id, Record.backtrace).join(Classification)
|
||||
q = q.filter(Classification.classification.like('org.clearlinux/swupd-client/%'))
|
||||
q = db.session.query(Record.timestamp_client, Record.machine_id, Record.payload)
|
||||
q = q.filter(Record.classification.like('org.clearlinux/swupd-client/%'))
|
||||
|
||||
if most_recent:
|
||||
interval_sec = 24 * 60 * 60 * int(most_recent)
|
||||
current_time = time()
|
||||
sec_in_past = current_time - interval_sec
|
||||
q = q.filter(Record.tsp > sec_in_past)
|
||||
q = q.filter(Record.timestamp_client > sec_in_past)
|
||||
|
||||
q = q.order_by(desc(Record.tsp))
|
||||
q = q.order_by(desc(Record.timestamp_client))
|
||||
return q
|
||||
|
||||
@staticmethod
|
||||
@@ -568,19 +529,18 @@ class Record(db.Model):
|
||||
internal_expr = case([(Record.external == False, Record.machine_id), ]).label('internal_count')
|
||||
external_expr = case([(Record.external == True, Record.machine_id), ]).label('external_count')
|
||||
|
||||
q = db.session.query(Build.build, db.func.count(db.distinct(internal_expr)), db.func.count(db.distinct(external_expr)))
|
||||
q = q.join(Record).join(Classification)
|
||||
q = q.filter(Classification.classification == "org.clearlinux/heartbeat/ping")
|
||||
q = q.filter(Record.os_name == 'clear-linux-os')
|
||||
q = q.group_by(Build.build)
|
||||
q = db.session.query(Record.build, db.func.count(db.distinct(internal_expr)), db.func.count(db.distinct(external_expr)))
|
||||
q = q.filter(Record.classification == "org.clearlinux/heartbeat/ping")
|
||||
q = q.filter(Record.system_name == 'clear-linux-os')
|
||||
q = q.group_by(Record.build)
|
||||
|
||||
if most_recent:
|
||||
interval_sec = 24 * 60 * 60 * int(most_recent)
|
||||
current_time = time()
|
||||
sec_in_past = current_time - interval_sec
|
||||
q = q.filter(Record.tsp > sec_in_past)
|
||||
q = q.filter(Record.timestamp_client > sec_in_past)
|
||||
|
||||
q = q.order_by(cast(Build.build, db.Integer))
|
||||
q = q.order_by(cast(Record.build, db.Integer))
|
||||
return q.all()
|
||||
|
||||
|
||||
@@ -656,5 +616,4 @@ class GuiltyBlacklist(db.Model):
|
||||
db.session.rollback()
|
||||
raise
|
||||
|
||||
|
||||
# vi: ts=4 et sw=4 sts=4
|
||||
|
||||
@@ -0,0 +1,341 @@
|
||||
"""empty message
|
||||
|
||||
Revision ID: 18e790856b53
|
||||
Revises: cb72058cf4e4
|
||||
Create Date: 2018-07-20 02:51:24.952915
|
||||
|
||||
"""
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '18e790856b53'
|
||||
down_revision = 'cb72058cf4e4'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
def upgrade():
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
|
||||
connection = op.get_bind()
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS machine_ids_24h CASCADE
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS top_builds CASCADE
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS crashes CASCADE
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS record_stream CASCADE
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS population_external CASCADE
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS population_internal CASCADE
|
||||
""")
|
||||
|
||||
with op.batch_alter_table('records') as bop:
|
||||
bop.alter_column('record_format_version', new_column_name='record_version')
|
||||
bop.alter_column('payload_format_version', new_column_name='payload_version')
|
||||
bop.alter_column('tsp', new_column_name='timestamp_client', type_=sa.NUMERIC())
|
||||
bop.alter_column('tsp_server', new_column_name='timestamp_server', type_=sa.NUMERIC(), nullable=False,
|
||||
server_default=sa.text("(date_part('epoch'::text, timezone('utc'::text, now())))::integer"))
|
||||
bop.alter_column('os_name', new_column_name='system_name', type_=sa.TEXT())
|
||||
bop.alter_column('machine', new_column_name='host_type')
|
||||
bop.alter_column('payload', type_=sa.TEXT(), nullable=False)
|
||||
bop.alter_column('bios_version', type_=sa.TEXT())
|
||||
bop.alter_column('board_name', type_=sa.TEXT())
|
||||
bop.alter_column('cpu_model', type_=sa.TEXT())
|
||||
bop.alter_column('event_id', type_=sa.TEXT())
|
||||
op.add_column('records', sa.Column('build', sa.TEXT(), nullable=False, server_default=''))
|
||||
op.add_column('records', sa.Column('classification', sa.TEXT(), nullable=False, server_default=''))
|
||||
op.drop_column('records', 'dupe_of')
|
||||
op.drop_column('records', 'icon')
|
||||
op.drop_column('records', 'buildstamp')
|
||||
op.drop_column('records', 'hide')
|
||||
op.drop_column('records', 'dupecount')
|
||||
op.drop_column('records', 'security')
|
||||
op.drop_column('records', 'dupemaster')
|
||||
|
||||
connection.execute("""
|
||||
UPDATE records SET build = build.build FROM build WHERE build.id = records.build_id
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
UPDATE records SET classification = classification.classification FROM classification WHERE classification.id = records.classification_id
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
UPDATE records SET payload = backtrace
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW machine_ids_24h AS
|
||||
(
|
||||
SELECT records.machine_id
|
||||
FROM records
|
||||
WHERE records.timestamp_server::double precision >= (date_part('epoch'::text, now()) - (24 * 3600)::double precision)
|
||||
GROUP BY records.machine_id
|
||||
)
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW top_builds AS
|
||||
(
|
||||
SELECT a.build::numeric AS build_no
|
||||
FROM (SELECT DISTINCT(records.build) FROM records WHERE records.build::numeric < 100000::numeric) a
|
||||
ORDER BY (a.build::numeric) DESC
|
||||
LIMIT 10
|
||||
)
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW crashes AS
|
||||
(
|
||||
SELECT g.function,
|
||||
count(g.function) AS count,
|
||||
r.build
|
||||
FROM records r,
|
||||
guilty g
|
||||
WHERE r.guilty_id = g.id
|
||||
AND (r.classification::text = ANY (ARRAY['org.clearlinux/crash/clr'::character varying::text, 'org.clearlinux/kernel/bug'::character varying::text, 'org.clearlinux/kernel/stackoverflow'::character varying::text, 'org.clearlinux/kernel/warning'::character varying::text]))
|
||||
AND r.build::numeric < 100000::numeric
|
||||
AND r.build::numeric >
|
||||
(
|
||||
SELECT top_builds.build_no
|
||||
FROM top_builds
|
||||
OFFSET 9
|
||||
LIMIT 1
|
||||
)
|
||||
GROUP BY g.function, r.build
|
||||
)
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW record_stream AS
|
||||
(
|
||||
SELECT to_timestamp(records.timestamp_server::double precision) AS to_timestamp,
|
||||
records.external,
|
||||
CASE
|
||||
WHEN records.external = true THEN 'external'::text
|
||||
ELSE 'internal'::text
|
||||
END AS source,
|
||||
records.machine_id,
|
||||
records.severity,
|
||||
records.classification,
|
||||
records.build
|
||||
FROM records
|
||||
ORDER BY records.id DESC
|
||||
LIMIT 1000
|
||||
)
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW population_external AS
|
||||
(
|
||||
SELECT 1 AS value,
|
||||
c.build AS metric,
|
||||
c.time
|
||||
FROM (
|
||||
SELECT max(a.build) AS build,
|
||||
max(a.timestamp_server) AS time
|
||||
FROM (
|
||||
SELECT records.machine_id,
|
||||
records.build,
|
||||
records.timestamp_server AS timestamp_server
|
||||
FROM records
|
||||
WHERE records.timestamp_server::double precision >= (date_part('epoch'::text, now()) - (24 * 3600)::double precision)
|
||||
AND records.external = true
|
||||
AND records.build::numeric < 100000::numeric
|
||||
AND records.build::numeric > 12000::numeric
|
||||
) a
|
||||
GROUP BY a.machine_id
|
||||
) c
|
||||
)
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW population_internal AS
|
||||
(
|
||||
SELECT 1 AS value,
|
||||
c.build AS metric,
|
||||
c.time
|
||||
FROM (
|
||||
SELECT max(a.build) AS build,
|
||||
max(a.timestamp_server) AS time
|
||||
FROM (
|
||||
SELECT records.machine_id,
|
||||
records.build,
|
||||
records.timestamp_server AS timestamp_server
|
||||
FROM records
|
||||
WHERE records.timestamp_server::double precision >= (date_part('epoch'::text, now()) - (24 * 3600)::double precision)
|
||||
AND records.external = false
|
||||
AND records.build::numeric < 100000::numeric
|
||||
AND records.build::numeric > 12000::numeric
|
||||
) a
|
||||
GROUP BY a.machine_id
|
||||
) c
|
||||
)
|
||||
""")
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade():
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
|
||||
connection = op.get_bind()
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS machine_ids_24h CASCADE
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS top_builds CASCADE
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS crashes CASCADE
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS record_stream CASCADE
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS population_external CASCADE
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
DROP VIEW IF EXISTS population_internal CASCADE
|
||||
""")
|
||||
|
||||
with op.batch_alter_table('records') as bop:
|
||||
bop.alter_column('record_version', new_column_name='record_format_version')
|
||||
bop.alter_column('payload_version', new_column_name='payload_format_version')
|
||||
bop.alter_column('timestamp_client', new_column_name='tsp', type_=sa.INTEGER())
|
||||
bop.alter_column('timestamp_server', new_column_name='tsp_server', type_=sa.INTEGER())
|
||||
bop.alter_column('system_name', new_column_name='os_name')
|
||||
bop.alter_column('host_type', new_column_name='machine')
|
||||
#bop.alter_column('payload', type_=sa.LargeBinary(), nullable=True)
|
||||
op.drop_column('records', 'classification')
|
||||
op.drop_column('records', 'build')
|
||||
op.add_column('records', sa.Column('dupemaster', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True))
|
||||
op.add_column('records', sa.Column('security', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True))
|
||||
op.add_column('records', sa.Column('dupecount', sa.INTEGER(), server_default=sa.text('0'), autoincrement=False, nullable=True))
|
||||
op.add_column('records', sa.Column('hide', sa.BOOLEAN(), server_default=sa.text('false'), autoincrement=False, nullable=True))
|
||||
op.add_column('records', sa.Column('buildstamp', sa.TEXT(), server_default=sa.text("''::text"), autoincrement=False, nullable=True))
|
||||
op.add_column('records', sa.Column('icon', sa.TEXT(), server_default=sa.text("''::text"), autoincrement=False, nullable=True))
|
||||
op.add_column('records', sa.Column('dupe_of', sa.INTEGER(), server_default=sa.text('0'), autoincrement=False, nullable=True))
|
||||
|
||||
connection.execute("""
|
||||
ALTER TABLE records ALTER COLUMN payload TYPE BYTEA USING convert_to(payload, 'LATIN1')
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW machine_ids_24h AS
|
||||
(
|
||||
SELECT records.machine_id
|
||||
FROM records
|
||||
WHERE records.tsp_server::double precision >= (date_part('epoch'::text, now()) - (24 * 3600)::double precision)
|
||||
GROUP BY records.machine_id
|
||||
)
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW top_builds AS
|
||||
(
|
||||
SELECT build.id,
|
||||
build.build::numeric AS build_no
|
||||
FROM build
|
||||
WHERE build.build::numeric < 100000::numeric
|
||||
ORDER BY (build.build::numeric) DESC
|
||||
LIMIT 10
|
||||
)
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW crashes AS
|
||||
(
|
||||
SELECT g.function,
|
||||
count(g.function) AS count,
|
||||
b.build
|
||||
FROM records,
|
||||
guilty g,
|
||||
build b,
|
||||
classification c
|
||||
WHERE records.guilty_id = g.id AND records.classification_id = c.id AND (c.classification::text = ANY (ARRAY['org.clearlinux/crash/clr'::character varying::text, 'org.clearlinux/kernel/bug'::character varying::text, 'org.clearlinux/kernel/stackoverflow'::character varying::text, 'org.clearlinux/kernel/warning'::character varying::text])) AND b.id = records.build_id AND b.build::numeric < 100000::numeric AND b.build::numeric > (( SELECT top_builds.build_no
|
||||
FROM top_builds
|
||||
OFFSET 9
|
||||
LIMIT 1))
|
||||
GROUP BY g.function, b.build
|
||||
)
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW record_stream AS
|
||||
(
|
||||
SELECT to_timestamp(records.tsp_server::double precision) AS to_timestamp,
|
||||
records.external,
|
||||
CASE
|
||||
WHEN records.external = true THEN 'external'::text
|
||||
ELSE 'internal'::text
|
||||
END AS source,
|
||||
records.machine_id,
|
||||
records.severity,
|
||||
c.classification,
|
||||
b.build
|
||||
FROM records,
|
||||
classification c,
|
||||
build b
|
||||
WHERE c.id = records.classification_id AND b.id = records.build_id
|
||||
ORDER BY records.id DESC
|
||||
LIMIT 1000
|
||||
)
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW population_external AS
|
||||
(
|
||||
SELECT 1 AS value,
|
||||
b.build AS metric,
|
||||
c."time"
|
||||
FROM ( SELECT max(a.build_id) AS build_id,
|
||||
max(a.tsp_server) AS "time"
|
||||
FROM ( SELECT records.machine_id,
|
||||
records.build_id,
|
||||
records.tsp_server
|
||||
FROM records
|
||||
WHERE records.tsp_server::double precision >= (date_part('epoch'::text, now()) - (24 * 3600)::double precision) AND records.external = true) a
|
||||
GROUP BY a.machine_id) c,
|
||||
build b
|
||||
WHERE c.build_id = b.id AND b.build::numeric < 100000::numeric AND b.build::numeric > 12000::numeric
|
||||
)
|
||||
""")
|
||||
|
||||
connection.execute("""
|
||||
CREATE VIEW population_internal AS
|
||||
(
|
||||
SELECT 1 AS value,
|
||||
b.build AS metric,
|
||||
c."time"
|
||||
FROM ( SELECT max(a.build_id) AS build_id,
|
||||
max(a.tsp_server) AS "time"
|
||||
FROM ( SELECT records.machine_id,
|
||||
records.build_id,
|
||||
records.tsp_server
|
||||
FROM records
|
||||
WHERE records.tsp_server::double precision >= (date_part('epoch'::text, now()) - (24 * 3600)::double precision) AND records.external = false) a
|
||||
GROUP BY a.machine_id) c,
|
||||
build b
|
||||
WHERE c.build_id = b.id AND b.build::numeric < 100000::numeric AND b.build::numeric > 12000::numeric
|
||||
)
|
||||
""")
|
||||
# ### end Alembic commands ###
|
||||
@@ -21,7 +21,7 @@ class Config(object):
|
||||
DEBUG = False
|
||||
TESTING = False
|
||||
LOG_LEVEL = logging.ERROR
|
||||
SQLALCHEMY_DATABASE_URI = 'postgres://postgres:@@db_password@@@localhost/telemdb'
|
||||
SQLALCHEMY_DATABASE_URI = 'postgres://postgres:@@db_password@@@localhost/telemetry'
|
||||
SQLALCHEMY_TRACK_MODIFICATIONS = True
|
||||
LOG_FILE = 'handler.log'
|
||||
WTF_CSRF_ENABLED = True
|
||||
|
||||
@@ -25,8 +25,8 @@ class Config(object):
|
||||
TESTING = False
|
||||
LOG_LEVEL = logging.ERROR
|
||||
|
||||
# If your telemdb database password is not 'postgres', update this line.
|
||||
SQLALCHEMY_DATABASE_URI = 'postgres://postgres:postgres@localhost/telemdb'
|
||||
# If your telemetry database password is not 'postgres', update this line.
|
||||
SQLALCHEMY_DATABASE_URI = 'postgres://postgres:postgres@localhost/telemetry'
|
||||
|
||||
SQLALCHEMY_TRACK_MODIFICATIONS = True
|
||||
LOG_FILE = 'handler.log'
|
||||
|
||||
@@ -33,7 +33,7 @@ from wtforms.widgets import (
|
||||
|
||||
|
||||
class RecordFilterForm(FlaskForm):
|
||||
os_name = SelectField('OS Name', coerce=str, validators=[InputRequired()])
|
||||
system_name = SelectField('OS Name', coerce=str, validators=[InputRequired()])
|
||||
build = SelectField('Version', coerce=str, validators=[InputRequired()])
|
||||
classification = SelectField('Classification', coerce=str, validators=[InputRequired()])
|
||||
severity = SelectField('Severity', coerce=str, validators=[InputRequired()])
|
||||
|
||||
@@ -118,7 +118,7 @@ class Config(object):
|
||||
DEBUG = True
|
||||
TESTING = False
|
||||
LOG_LEVEL = logging.ERROR
|
||||
SQLALCHEMY_DATABASE_URI = 'postgres://postgres:postgres@localhost/telemdb'
|
||||
SQLALCHEMY_DATABASE_URI = 'postgres://postgres:postgres@localhost/telemetry'
|
||||
SQLALCHEMY_TRACK_MODIFICATIONS = True
|
||||
LOG_FILE = 'telemetryui.log'
|
||||
WTF_CSRF_ENABLED = True
|
||||
|
||||
@@ -29,10 +29,10 @@
|
||||
<dd><a href="{{ url_for('records', page=1, machine_id=record.machine_id) }}">{{ record.machine_id }}</a></dd>
|
||||
|
||||
<dt>Machine Type</dt>
|
||||
<dd>{{ record.machine }}</dd>
|
||||
<dd>{{ record.host_type }}</dd>
|
||||
|
||||
<dt>Operating System</dt>
|
||||
<dd>{{ record.os_name }}</dd>
|
||||
<dd>{{ record.system_name }}</dd>
|
||||
|
||||
<dt>Kernel Version</dt>
|
||||
<dd>{{ record.kernel_version }}</dd>
|
||||
@@ -50,19 +50,19 @@
|
||||
<dd>{{ record.cpu_model if record.cpu_model else 'N/A'}}</dd>
|
||||
|
||||
<dt>Record Time</dt>
|
||||
<dd>{{ record.tsp|local_datetime_since }}</dd>
|
||||
<dd>{{ record.timestamp_client|local_datetime_since }}</dd>
|
||||
|
||||
<dt>Server Time</dt>
|
||||
<dd>{{ record.tsp_server|local_datetime_since }} </dd>
|
||||
<dd>{{ record.timestamp_server|local_datetime_since }} </dd>
|
||||
|
||||
<dt>Build</dt>
|
||||
<dd><a href="{{ url_for('records', page=1, build=record.build.build) }}">{{ record.build.build }}</a></dd>
|
||||
<dd><a href="{{ url_for('records', page=1, build=record.build) }}">{{ record.build }}</a></dd>
|
||||
|
||||
<dt>Severity</dt>
|
||||
<dd><a href="{{ url_for('records', page=1, severity=record.severity) }}">{{ record.severity }}</a></dd>
|
||||
|
||||
<dt>Classification</dt>
|
||||
<dd><a id="class_link" href="{{ url_for('records', page=1, classification=record.classification.classification) }}">{{ record.classification.classification }}</a></dd>
|
||||
<dd><a id="class_link" href="{{ url_for('records', page=1, classification=record.classification) }}">{{ record.classification }}</a></dd>
|
||||
|
||||
<dt>Guilty Id</dt>
|
||||
<dd>
|
||||
@@ -73,14 +73,14 @@
|
||||
{%- endif %}
|
||||
</dd>
|
||||
|
||||
<dt>Payload Format Version</dt>
|
||||
<dd>{{ record.payload_format_version }}</dd>
|
||||
<dt>Payload Version</dt>
|
||||
<dd>{{ record.payload_version }}</dd>
|
||||
|
||||
<dt>External</dt>
|
||||
<dd>{{ record.external }}</dd>
|
||||
|
||||
<dt>Payload</dt>
|
||||
<dd> <pre class="prettyprint">{{ record.backtrace }}</pre></dd>
|
||||
<dd> <pre class="prettyprint">{{ record.payload }}</pre></dd>
|
||||
|
||||
</dl>
|
||||
{% endblock %}
|
||||
|
||||
@@ -22,8 +22,8 @@
|
||||
{{ form.hidden_tag() }}
|
||||
<div class="form-group row">
|
||||
<div class="col-sm-3">
|
||||
{{ form.os_name.label(class="control-label", for="os_name") }}
|
||||
{{ form.os_name(id="os_name", class="form-control") }}
|
||||
{{ form.system_name.label(class="control-label", for="system_name") }}
|
||||
{{ form.system_name(id="system_name", class="form-control") }}
|
||||
</div>
|
||||
<div class="col-sm-3" >
|
||||
{{ form.build.label(class="control-label" , for="build_id") }}
|
||||
@@ -86,11 +86,11 @@
|
||||
}
|
||||
});
|
||||
}
|
||||
$(document).on('change', 'select#os_name', function () {
|
||||
$(document).on('change', 'select#system_name', function () {
|
||||
$('select#build_id').val("All")
|
||||
selectBuilds($(this).val());
|
||||
});
|
||||
selectBuilds($('select#os_name').val());
|
||||
selectBuilds($('select#system_name').val());
|
||||
</script>
|
||||
{% endblock %}
|
||||
{% include "records_template.html" %}
|
||||
|
||||
@@ -41,14 +41,14 @@
|
||||
<td>internal</td>
|
||||
{%- endif %}
|
||||
<td><span data-toggle="tooltip" data-placement="top" title="{{ rec.machine_id }}"><div class="machine_id">{{ rec.machine_id }}</div></span></td>
|
||||
<td>{{ rec.tsp_server | timesince }}</td>
|
||||
<td>{{ rec.timestamp_server | int | timesince }}</td>
|
||||
<td><span class="label label-{{ (rec.severity | get_severity_label).0 }}">{{ (rec.severity | get_severity_label).1 }}</span></td>
|
||||
<td class="ellipsize">{{ rec.classification.classification|replace("org.clearlinux/", "") }}</td>
|
||||
<td>{{ rec.os_name }}</td>
|
||||
<td>{{ rec.build.build }}</td>
|
||||
<td class="ellipsize row-payload">{{ rec.backtrace }}</td>
|
||||
<td class="ellipsize">{{ rec.classification|replace("org.clearlinux/", "") }}</td>
|
||||
<td>{{ rec.system_name }}</td>
|
||||
<td>{{ rec.build }}</td>
|
||||
<td class="ellipsize row-payload">{{ rec.payload }}</td>
|
||||
<td>
|
||||
<button id="more_button" type="button" data-toggle="modal" data-target="#myModal" data-whatever="{{ rec.backtrace }}">...</button>
|
||||
<button id="more_button" type="button" data-toggle="modal" data-target="#myModal" data-whatever="{{ rec.payload }}">...</button>
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
|
||||
@@ -62,7 +62,7 @@ def records(page=1):
|
||||
cl = session.get('severity')
|
||||
|
||||
os_map = Record.get_os_map()
|
||||
form.os_name.choices = [("All", "All")] + [(n, n) for n in os_map.keys()]
|
||||
form.system_name.choices = [("All", "All")] + [(n, n) for n in os_map.keys()]
|
||||
|
||||
form.machine_id.default = ""
|
||||
|
||||
@@ -81,7 +81,7 @@ def records(page=1):
|
||||
return render_template('records.html', records=out_records, form=form)
|
||||
else:
|
||||
classification = request.form.get('classification')
|
||||
os_name = request.form.get('os_name')
|
||||
system_name = request.form.get('system_name')
|
||||
build = request.form.get('build')
|
||||
severity = request.form.get('severity')
|
||||
page_size = request.form.get('page_size')
|
||||
@@ -94,7 +94,7 @@ def records(page=1):
|
||||
|
||||
redirect_args = {
|
||||
"page_size": page_size if page_size != "" else None,
|
||||
"os_name": os_name if os_name != "All" else None,
|
||||
"system_name": system_name if system_name != "All" else None,
|
||||
"build": build if build != "All" else None,
|
||||
"severity": severity if severity != "All" else None,
|
||||
"classification": classification if classification != "All" else None,
|
||||
@@ -122,7 +122,7 @@ def records(page=1):
|
||||
|
||||
elif request.method == 'GET':
|
||||
classification = request.args.get('classification')
|
||||
os_name = request.args.get('os_name')
|
||||
system_name = request.args.get('system_name')
|
||||
build = request.args.get('build')
|
||||
severity = request.args.get('severity')
|
||||
page_size = request.args.get('page_size')
|
||||
@@ -137,8 +137,8 @@ def records(page=1):
|
||||
form.classification.default = classification
|
||||
if build is not None:
|
||||
form.build.default = build
|
||||
if os_name is not None:
|
||||
form.os_name.default = os_name
|
||||
if system_name is not None:
|
||||
form.system_name.default = system_name
|
||||
if severity is not None:
|
||||
form.severity.default = severity
|
||||
if machine_id is not None:
|
||||
@@ -160,7 +160,7 @@ def records(page=1):
|
||||
page_size = RECORDS_PER_PAGE
|
||||
elif int(page_size) > MAX_RECORDS_PER_PAGE:
|
||||
page_size = MAX_RECORDS_PER_PAGE
|
||||
out_records = Record.filter_records(build, classification, severity, machine_id, os_name=os_name,
|
||||
out_records = Record.filter_records(build, classification, severity, machine_id, system_name=system_name,
|
||||
payload=payload, not_payload=not_payload, data_source=data_source,
|
||||
from_date=from_date, to_date=to_date).paginate(page, int(page_size), False)
|
||||
return render_template('records.html', records=out_records, form=form, os_map=json.dumps(os_map))
|
||||
@@ -466,17 +466,17 @@ def mce():
|
||||
week_rec_map = {}
|
||||
class_rec_map = {}
|
||||
for record in records:
|
||||
week = time.strftime("%U", time.localtime(int(record.tsp)))
|
||||
week_rec_map.setdefault(record.classification.classification, {}).setdefault(week, 0)
|
||||
week_rec_map[record.classification.classification][week] += 1
|
||||
class_rec_map.setdefault(record.classification.classification, 0)
|
||||
class_rec_map[record.classification.classification] += 1
|
||||
week = time.strftime("%U", time.localtime(int(record.timestamp_client)))
|
||||
week_rec_map.setdefault(record.classification, {}).setdefault(week, 0)
|
||||
week_rec_map[record.classification][week] += 1
|
||||
class_rec_map.setdefault(record.classification, 0)
|
||||
class_rec_map[record.classification] += 1
|
||||
by_machine_id.setdefault(record.machine_id, {"builds": {}, "recordscnt": 0})
|
||||
by_machine_id[record.machine_id]["builds"].setdefault(record.build.build, 0)
|
||||
by_machine_id[record.machine_id]["builds"][record.build.build] += 1
|
||||
by_machine_id[record.machine_id]["builds"].setdefault(record.build, 0)
|
||||
by_machine_id[record.machine_id]["builds"][record.build] += 1
|
||||
by_machine_id[record.machine_id]["recordscnt"] += 1
|
||||
by_builds.setdefault(record.build.build, 0)
|
||||
by_builds[record.build.build] += 1
|
||||
by_builds.setdefault(record.build, 0)
|
||||
by_builds[record.build] += 1
|
||||
for machine_id in list(by_machine_id.keys()):
|
||||
builds_cnt = by_machine_id[machine_id]["builds"]
|
||||
maxcnt = max(list(builds_cnt.values()) + [maxcnt])
|
||||
@@ -501,8 +501,8 @@ def thermal():
|
||||
week_rec_map = {}
|
||||
year_start = time.mktime(time.strptime(current_year + "000", "%Y%U%w"))
|
||||
for r in records:
|
||||
if r.tsp >= year_start:
|
||||
week = str(int((r.tsp - year_start) / 604800) + ((r.tsp - year_start) % 604800 > 0))
|
||||
if r.timestamp_client >= year_start:
|
||||
week = str(int((r.timestamp_client - year_start) / 604800) + ((r.timestamp_client - year_start) % 604800 > 0))
|
||||
if week in week_rec_map:
|
||||
week_rec_map[week] += 1
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user