Send logs from your product

Course

This procedure is a part of a course that teaches you how to build a quickstart. If you haven't already, checkout the course introduction.

Each procedure in this course builds on top of the last one, so make sure you've completed the last procedure, send events from your product before proceeding with this one.

Logs are generated by applications. They are time-based text records that help your users see what's happening in your system.

New Relic, provides you a variety of ways to instrument your application to send logs to our Logs API.

In this lesson, you learn to send logs from your product using our telemetry software development kit (SDK).

Use our SDK

We offer an open source telemetry SDK in several of the most popular programming languages. These send data to our data ingest APIs, including our Log API. Of these language SDKs, Python and Java work with the Log API.

In this lesson, you learn how to install and use the Python telemetry SDK to send logs to New Relic.

Step 1 of 7

Change to the send-logs/flashDB direcrory of the course repository.

bash

$cd ../../send-events/flashDB

Step 2 of 7

If you haven't already, install the newrelic-telemetry-sdk package.

bash

$pip install newrelic-telemetry-sdk

Step 3 of 7

Open db.py file in the IDE of your choice and configure the LogClient.

1
import os
2
import random
3
import datetime
4
from sys import getsizeof
5

6
from newrelic_telemetry_sdk import MetricClient, GaugeMetric, CountMetric, SummaryMetric
7
from newrelic_telemetry_sdk import EventClient, Event
8
from newrelic_telemetry_sdk import LogClient
9

10
metric_client = MetricClient(os.environ["NEW_RELIC_LICENSE_KEY"])
11
event_client = EventClient(os.environ["NEW_RELIC_LICENSE_KEY"])
12
log_client = LogClient(os.environ["NEW_RELIC_LICENSE_KEY"])
13

14
db = {}
15
stats = {
16
    "read_response_times": [],
17
    "read_errors": 0,
18
    "read_count": 0,
19
    "create_response_times": [],
20
    "create_errors": 0,
21
    "create_count": 0,
22
    "update_response_times": [],
23
    "update_errors": 0,
24
    "update_count": 0,
25
    "delete_response_times": [],
26
    "delete_errors": 0,
27
    "delete_count": 0,
28
    "cache_hit": 0,
29
}
30
last_push = {
31
    "read": datetime.datetime.now(),
32
    "create": datetime.datetime.now(),
33
    "update": datetime.datetime.now(),
34
    "delete": datetime.datetime.now(),
35
}
36

37
def read(key):
38

39
    print(f"Reading...")
40

41
    if random.randint(0, 30) > 10:
42
        stats["cache_hit"] += 1
43

44
    stats["read_response_times"].append(random.uniform(0.5, 1.0))
45
    if random.choice([True, False]):
46
        stats["read_errors"] += 1
47
    stats["read_count"] += 1
48
    try_send("read")
49

50
def create(key, value):
51

52
    print(f"Writing...")
53

54
    db[key] = value
55
    stats["create_response_times"].append(random.uniform(0.5, 1.0))
56
    if random.choice([True, False]):
57
        stats["create_errors"] += 1
58
    stats["create_count"] += 1
59
    try_send("create")
60

61
def update(key, value):
62

63
    print(f"Updating...")
64

65
    db[key] = value
66
    stats["update_response_times"].append(random.uniform(0.5, 1.0))
67
    if random.choice([True, False]):
68
        stats["update_errors"] += 1
69
    stats["update_count"] += 1
70
    try_send("update")
71

72
def delete(key):
73

74
    print(f"Deleting...")
75

76
    db.pop(key, None)
77
    stats["delete_response_times"].append(random.uniform(0.5, 1.0))
78
    if random.choice([True, False]):
79
        stats["delete_errors"] += 1
80
    stats["delete_count"] += 1
81
    try_send("delete")
82

83
def try_send(type_):
84

85
    print("try_send")
86

87
    now = datetime.datetime.now()
88
    interval_ms = (now - last_push[type_]).total_seconds() * 1000
89
    if interval_ms >= 2000:
90
        send_metrics(type_, interval_ms)
91
        send_event(type_)
92

93
def send_metrics(type_, interval_ms):
94
    
95
    print("sending metrics...")
96

97
    keys = GaugeMetric("fdb_keys", len(db))
98
    db_size = GaugeMetric("fdb_size", getsizeof(db))
99

100
    errors = CountMetric(
101
        name=f"fdb_{type_}_errors",
102
        value=stats[f"{type_}_errors"],
103
        interval_ms=interval_ms
104
    )
105

106
    cache_hits = CountMetric(
107
        name=f"fdb_cache_hits",
108
        value=stats["cache_hit"],
109
        interval_ms=interval_ms
110
    )
111

112
    response_times = stats[f"{type_}_response_times"]
113
    response_time_summary = SummaryMetric(
114
        f"fdb_{type_}_responses",
115
        count=len(response_times),
116
        min=min(response_times),
117
        max=max(response_times),
118
        sum=sum(response_times),
119
        interval_ms=interval_ms,
120
    )
121

122
    batch = [keys, db_size, errors, cache_hits, response_time_summary]
123
    response = metric_client.send_batch(batch)
124
    response.raise_for_status()
125
    print("Sent metrics successfully!")
126
    clear(type_)
127

128
def send_event(type_):
129

130
    print("sending event...")
131

132
    count = Event(
133
        "fdb_method", {"method": type_}
134
    )
135

136
    response = event_client.send_batch(count)
137
    response.raise_for_status()
138
    print("Event sent successfully!")
139

140
def clear(type_):
141
    stats[f"{type_}_response_times"] = []
142
    stats[f"{type_}_errors"] = 0
143
    stats["cache_hit"] = 0
144
    stats[f"{type_}_count"] = 0
145
    last_push[type_] = datetime.datetime.now()

db.py

Important

This example expects an environment variable called $NEW_RELIC_LICENSE_KEY.

Step 4 of 7

Instrument your app to send logs to New Relic.

1
import os
2
import random
3
import datetime
4
from sys import getsizeof
5
import psutil
6

7
from newrelic_telemetry_sdk import MetricClient, GaugeMetric, CountMetric, SummaryMetric
8
from newrelic_telemetry_sdk import EventClient, Event
9
from newrelic_telemetry_sdk import LogClient, Log
10

11
metric_client = MetricClient(os.environ["NEW_RELIC_LICENSE_KEY"])
12
event_client = EventClient(os.environ["NEW_RELIC_LICENSE_KEY"])
13
log_client = LogClient(os.environ["NEW_RELIC_LICENSE_KEY"])
14

15
db = {}
16
stats = {
17
    "read_response_times": [],
18
    "read_errors": 0,
19
    "read_count": 0,
20
    "create_response_times": [],
21
    "create_errors": 0,
22
    "create_count": 0,
23
    "update_response_times": [],
24
    "update_errors": 0,
25
    "update_count": 0,
26
    "delete_response_times": [],
27
    "delete_errors": 0,
28
    "delete_count": 0,
29
    "cache_hit": 0,
30
}
31
last_push = {
32
    "read": datetime.datetime.now(),
33
    "create": datetime.datetime.now(),
34
    "update": datetime.datetime.now(),
35
    "delete": datetime.datetime.now(),
36
}
37

38
def read(key):
39

40
    print(f"Reading...")
41

42
    if random.randint(0, 30) > 10:
43
        stats["cache_hit"] += 1
44

45
    stats["read_response_times"].append(random.uniform(0.5, 1.0))
46
    if random.choice([True, False]):
47
        stats["read_errors"] += 1
48
    stats["read_count"] += 1
49
    try_send("read")
50

51
def create(key, value):
52

53
    print(f"Writing...")
54

55
    db[key] = value
56
    stats["create_response_times"].append(random.uniform(0.5, 1.0))
57
    if random.choice([True, False]):
58
        stats["create_errors"] += 1
59
    stats["create_count"] += 1
60
    try_send("create")
61

62
def update(key, value):
63

64
    print(f"Updating...")
65

66
    db[key] = value
67
    stats["update_response_times"].append(random.uniform(0.5, 1.0))
68
    if random.choice([True, False]):
69
        stats["update_errors"] += 1
70
    stats["update_count"] += 1
71
    try_send("update")
72

73
def delete(key):
74

75
    print(f"Deleting...")
76

77
    db.pop(key, None)
78
    stats["delete_response_times"].append(random.uniform(0.5, 1.0))
79
    if random.choice([True, False]):
80
        stats["delete_errors"] += 1
81
    stats["delete_count"] += 1
82
    try_send("delete")
83

84
def try_send(type_):
85

86
    print("try_send")
87

88
    now = datetime.datetime.now()
89
    interval_ms = (now - last_push[type_]).total_seconds() * 1000
90
    if interval_ms >= 2000:
91
        send_metrics(type_, interval_ms)
92
        send_event(type_)
93

94
def send_metrics(type_, interval_ms):
95
    
96
    print("sending metrics...")
97

98
    keys = GaugeMetric("fdb_keys", len(db))
99
    db_size = GaugeMetric("fdb_size", getsizeof(db))
100

101
    errors = CountMetric(
102
        name=f"fdb_{type_}_errors",
103
        value=stats[f"{type_}_errors"],
104
        interval_ms=interval_ms
105
    )
106

107
    cache_hits = CountMetric(
108
        name=f"fdb_cache_hits",
109
        value=stats["cache_hit"],
110
        interval_ms=interval_ms
111
    )
112

113
    response_times = stats[f"{type_}_response_times"]
114
    response_time_summary = SummaryMetric(
115
        f"fdb_{type_}_responses",
116
        count=len(response_times),
117
        min=min(response_times),
118
        max=max(response_times),
119
        sum=sum(response_times),
120
        interval_ms=interval_ms,
121
    )
122

123
    batch = [keys, db_size, errors, cache_hits, response_time_summary]
124
    response = metric_client.send_batch(batch)
125
    response.raise_for_status()
126
    print("Sent metrics successfully!")
127
    clear(type_)
128

129
def send_event(type_):
130

131
    print("sending event...")
132

133
    count = Event(
134
        "fdb_method", {"method": type_}
135
    )
136

137
    response = event_client.send_batch(count)
138
    response.raise_for_status()
139
    print("Event sent successfully!")
140

141
def send_logs():
142

143
    print("sending log...")
144

145
    process = psutil.Process(os.getpid())
146
    memory_usage = process.memory_percent()
147

148
    log = Log("FlashDB is using " + str(round(memory_usage * 100, 2)) + "% memory")
149

150
    response = log_client.send(log)
151
    response.raise_for_status()
152
    print("Log sent successfully!")
153

154
def clear(type_):
155
    stats[f"{type_}_response_times"] = []
156
    stats[f"{type_}_errors"] = 0
157
    stats["cache_hit"] = 0
158
    stats[f"{type_}_count"] = 0
159
    last_push[type_] = datetime.datetime.now()

db.py

Here, you instrument your platform to send memory_usage as log to New Relic.

Step 5 of 7

Amend the try_send module to send the logs every 2 second.

1
import os
2
import random
3
import datetime
4
from sys import getsizeof
5
import psutil
6

7
from newrelic_telemetry_sdk import MetricClient, GaugeMetric, CountMetric, SummaryMetric
8
from newrelic_telemetry_sdk import EventClient, Event
9
from newrelic_telemetry_sdk import LogClient, Log
10

11
metric_client = MetricClient(os.environ["NEW_RELIC_LICENSE_KEY"])
12
event_client = EventClient(os.environ["NEW_RELIC_LICENSE_KEY"])
13
log_client = LogClient(os.environ["NEW_RELIC_LICENSE_KEY"])
14

15
db = {}
16
stats = {
17
    "read_response_times": [],
18
    "read_errors": 0,
19
    "read_count": 0,
20
    "create_response_times": [],
21
    "create_errors": 0,
22
    "create_count": 0,
23
    "update_response_times": [],
24
    "update_errors": 0,
25
    "update_count": 0,
26
    "delete_response_times": [],
27
    "delete_errors": 0,
28
    "delete_count": 0,
29
    "cache_hit": 0,
30
}
31
last_push = {
32
    "read": datetime.datetime.now(),
33
    "create": datetime.datetime.now(),
34
    "update": datetime.datetime.now(),
35
    "delete": datetime.datetime.now(),
36
}
37

38
def read(key):
39

40
    print(f"Reading...")
41

42
    if random.randint(0, 30) > 10:
43
        stats["cache_hit"] += 1
44

45
    stats["read_response_times"].append(random.uniform(0.5, 1.0))
46
    if random.choice([True, False]):
47
        stats["read_errors"] += 1
48
    stats["read_count"] += 1
49
    try_send("read")
50

51
def create(key, value):
52

53
    print(f"Writing...")
54

55
    db[key] = value
56
    stats["create_response_times"].append(random.uniform(0.5, 1.0))
57
    if random.choice([True, False]):
58
        stats["create_errors"] += 1
59
    stats["create_count"] += 1
60
    try_send("create")
61

62
def update(key, value):
63

64
    print(f"Updating...")
65

66
    db[key] = value
67
    stats["update_response_times"].append(random.uniform(0.5, 1.0))
68
    if random.choice([True, False]):
69
        stats["update_errors"] += 1
70
    stats["update_count"] += 1
71
    try_send("update")
72

73
def delete(key):
74

75
    print(f"Deleting...")
76

77
    db.pop(key, None)
78
    stats["delete_response_times"].append(random.uniform(0.5, 1.0))
79
    if random.choice([True, False]):
80
        stats["delete_errors"] += 1
81
    stats["delete_count"] += 1
82
    try_send("delete")
83

84
def try_send(type_):
85

86
    print("try_send")
87

88
    now = datetime.datetime.now()
89
    interval_ms = (now - last_push[type_]).total_seconds() * 1000
90
    if interval_ms >= 2000:
91
        send_metrics(type_, interval_ms)
92
        send_event(type_)
93
        send_logs()
94

95
def send_metrics(type_, interval_ms):
96
    
97
    print("sending metrics...")
98

99
    keys = GaugeMetric("fdb_keys", len(db))
100
    db_size = GaugeMetric("fdb_size", getsizeof(db))
101

102
    errors = CountMetric(
103
        name=f"fdb_{type_}_errors",
104
        value=stats[f"{type_}_errors"],
105
        interval_ms=interval_ms
106
    )
107

108
    cache_hits = CountMetric(
109
        name=f"fdb_cache_hits",
110
        value=stats["cache_hit"],
111
        interval_ms=interval_ms
112
    )
113

114
    response_times = stats[f"{type_}_response_times"]
115
    response_time_summary = SummaryMetric(
116
        f"fdb_{type_}_responses",
117
        count=len(response_times),
118
        min=min(response_times),
119
        max=max(response_times),
120
        sum=sum(response_times),
121
        interval_ms=interval_ms,
122
    )
123

124
    batch = [keys, db_size, errors, cache_hits, response_time_summary]
125
    response = metric_client.send_batch(batch)
126
    response.raise_for_status()
127
    print("Sent metrics successfully!")
128
    clear(type_)
129

130
def send_event(type_):
131

132
    print("sending event...")
133

134
    count = Event(
135
        "fdb_method", {"method": type_}
136
    )
137

138
    response = event_client.send_batch(count)
139
    response.raise_for_status()
140
    print("Event sent successfully!")
141

142
def send_logs():
143

144
    print("sending log...")
145

146
    process = psutil.Process(os.getpid())
147
    memory_usage = process.memory_percent()
148

149
    log = Log("FlashDB is using " + str(round(memory_usage * 100, 2)) + "% memory")
150

151
    response = log_client.send(log)
152
    response.raise_for_status()
153
    print("Log sent successfully!")
154

155
def clear(type_):
156
    stats[f"{type_}_response_times"] = []
157
    stats[f"{type_}_errors"] = 0
158
    stats["cache_hit"] = 0
159
    stats[f"{type_}_count"] = 0
160
    last_push[type_] = datetime.datetime.now()

db.py

Your platform will now report the configured logs every 2 seconds.

Step 6 of 7

Navigate to the root of your application at build-a-quickstart-lab/send-logs/flashDB.

Step 7 of 7

Run your services to verify that it is reporting logs.

bash

$python simulator.py
Writing...
try_send
Reading...
try_send
Reading...
try_send
Writing...
try_send
Writing...
try_send
Reading...
sending metrics...
Sent metrics successfully!
sending event...
Event sent successfully!
sending log...
Log sent successfully!

Alternative Options

If the language SDK doesn’t fit your needs, try out one of our other options:

New Relic offers a variety of log forwarding solutions that allow you to collect logs from operating systems, cloud platforms including Amazon AWS, [Google Cloud Platform], Microsoft Azure, and Heroku, Kubernetes, Docker, and APM.
Manual Implementation: If the previous options don’t fit your requirements, you can always manually instrument your own library to make a POST request to the New Relic Log API.

In this procedure, you instrumented your service to send logs to New Relic. Next, instrument it to send traces.

Course

This procedure is a part of course that teaches you how to build a quickstart. Continue to next lesson, send traces from your product.