Kafka logging with certificates from Vault PKI

volfagnt · April 27, 2022, 7:57pm

Hi all!
I would like to know if it is possible to send messages to the producer in k6 using logging with vault certificates in .pem format. I need to connect to the producer and write messages to the topic, but authorization happens with certificates. K6 can do that? If yes, then I will be grateful if anyone has an example of a connection.
Thanks.

mostafa · May 4, 2022, 4:03pm

Hi @volfagnt,

~~It doesn’t currently support SSL certs. Please keep an eye on this issue or help contribute to it, which is much appreciated.~~

The TLS support is already available, as documented in the JS API docs.

volfagnt · June 17, 2022, 5:37pm

Hi! I saw that changes were added to work with certificates. Can you tell if it only accepts certificates as a file or can it work with them as a string?
A have some errors with ssl and certs in .pem format

This Errors when i try create topic:

time=“2022-06-18T16:10:31+03:00” level=error msg=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})” error=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})”
time=“2022-06-18T16:10:31+03:00” level=error msg=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})” error=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})”
time=“2022-06-18T16:10:31+03:00” level=error msg=“Failed to create dialer., OriginalError: %!w(*fmt.wrapError=&{failed to dial: failed to open connection to [kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093]:9092: dial tcp: lookup kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093: no such host 0xc0026803c0})” error=“Failed to create dialer., OriginalError: %!w(*fmt.wrapError=&{failed to dial: failed to open connection to [kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093]:9092: dial tcp: lookup kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093: no such host 0xc0026803c0})”
time=“2022-06-18T16:10:31+03:00” level=error msg=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})” error=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})”

And when I use writer:
time=“2022-06-18T16:10:31+03:00” level=error msg=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})” error=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})”
time=“2022-06-18T16:10:31+03:00” level=error msg=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})” error=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})”
time=“2022-06-18T16:10:31+03:00” level=error msg=“Failed to create dialer., OriginalError: %!w(*fmt.wrapError=&{failed to dial: failed to open connection to [kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093]:9092: dial tcp: lookup kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093: no such host 0xc0026803c0})” error=“Failed to create dialer., OriginalError: %!w(*fmt.wrapError=&{failed to dial: failed to open connection to [kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093]:9092: dial tcp: lookup kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093: no such host 0xc0026803c0})”
time=“2022-06-18T16:10:31+03:00” level=error msg=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})” error="Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})"time=“2022-06-18T16:10:31+03:00” level=error msg=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})” error=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})”
time=“2022-06-18T16:10:31+03:00” level=error msg=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})” error=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})”
time=“2022-06-18T16:10:31+03:00” level=error msg=“Failed to create dialer., OriginalError: %!w(*fmt.wrapError=&{failed to dial: failed to open connection to [kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093]:9092: dial tcp: lookup kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093: no such host 0xc0026803c0})” error=“Failed to create dialer., OriginalError: %!w(*fmt.wrapError=&{failed to dial: failed to open connection to [kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093]:9092: dial tcp: lookup kafka-test-1:9093,kafka-test-2:9093,kafka-test-3:9093: no such host 0xc0026803c0})”
time=“2022-06-18T16:10:31+03:00” level=error msg=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})” error=“Unable to unmarshal credentials, OriginalError: %!w(*json.SyntaxError=&{invalid character ‘o’ looking for beginning of value 2})”

My code example:
const saslConfig = {
algorithm: “sasl_ssl”,
};

// TLS config is optional
const tlsConfig = {
// Enable/disable TLS (default: false)
enableTLS: true,
// Skip TLS verification if the certificate is invalid or self-signed (default: false)
insecureSkipTLSVerify: false,
// Possible values: “TLSv1.0”, “TLSv1.1”, “TLSv1.2” (default), “TLSv1.3”
minVersion: “TLSv1.0”,
// Only needed if you have a custom or self-signed certificate and keys
clientCertPem: “./client.pem”,
clientKeyPem: “./client-key.pem”,
serverCaPem: “./ca.pem”,
};

const [producer, _writerError] = writer(bootstrapServers, kafkaTopic, saslConfig, tlsConfig);
function getRandomArbitrary(min, max) {
return Math.random() * (max - min) + min;
}
for (let index = 0; index < 1; index++) {
var nf_k6_key = new Date().toISOString().slice(0, 10);
var currentDate = new Date().toISOString().slice(0, 19).replace(‘T’, ’ ');
var itm = getRandomArbitrary(1111111,7777777);
let messages = [
{
key: JSON.stringify({
correlationId: nf_k6_key + index,
}),
value: JSON.stringify({
“ITEMNO”: itm,
“SITEID”: “BLA-BLA.COM”,
“BUSPRODUCTID”: “”,
“CRMPRODUCTID”: “”,
“BUSSECTIONID”: “”,
“CRMSECTIONID”: “”,
“BODY”: {
“NAME”: " A4Tech Fstyler FB10C (fb10c matcha green NT)",
“GROUPCODE”: “PS020406”,
“EAN”: “”,
“CODE”: “www.bla-bla.com/a4tech-fb10c-” + itm
}
}),
headers: {
createdatutc: currentDate,
createdby: “SRV_NAV”,
aggregatetype:“40”,
},
},
];

     let error = produce(producer, messages);
     check(error, {
         "Messages are sent": (err) => err == undefined,
     });
 }

}

mostafa · June 20, 2022, 8:42am

@volfagnt It needs filenames. Which version of xk6-kafka are you using?

I made a few fixes very recently (below), which need to be built for testing because I haven’t released that code yet, due to the current release process. Also, the latest API changes are available here.

github.com/mostafa/xk6-kafka

Fix SASL and TLS issues reported in #56 and #84

mostafa:main ← mostafa:fix-sasl-and-tls-config

opened 03:20PM - 12 Jun 22 UTC

mostafa

+962 -275

In this PR, I tried to fix the issue reported in the following issues. - #56 -… #84 The PR includes changes, fixes, and features as follows: - **Fixes to how SASL and TLS configs work:** While refactoring the extension on several occasions over the past few months, I tried to improve the extension. Still, I somehow broke SASL and TLS configuration for the unauthenticated client (dialer) with the default TLS configuration (and introduced a regression). As I mentioned in https://github.com/mostafa/xk6-kafka/issues/56#issuecomment-1149750409 and https://github.com/mostafa/xk6-kafka/issues/56#issuecomment-1149798748, the problem manifested itself while a default TLS configuration was needed, but the extension was looking for certificate and keys to be passed explicitly. This is now fixed. - **Automatic topic creation:** I also introduced a new option to the `produce` function to allow [automatic topic creation](https://github.com/mostafa/xk6-kafka/pull/86/files#diff-86f23d296531dee41d5e2cf9ef90844267173280c625dbe2738cd163d2278168R58) on the first message. - **Passing values directly from JS to Go in functions as JSON:** I also experimented with [passing struct fields](https://github.com/mostafa/xk6-kafka/pull/86/files#diff-1f48700829d7e0cdba7f3729d6a7c1c5df3aa11f2aeeed583c7e1e39aa7b66d1R30-R43=) directly from JS to Go based on the [`json` field](https://github.com/mostafa/xk6-kafka/pull/86/files#diff-c95a9381cb3d755657cf984eba630f599695529c52a3b407d902d38bcea81f8aR47) in the Go struct, and it was successful 🎉, which means that I'll refactor the rest of the API to accept JSON, instead of stringified JSON. This was an effort started by @iamelevich in #20, but I intervened and made the PR a mess. 🤦 So, to make amends, I'll refactor all the APIs to receive JSON to make things easier, both for developers and users/testers. - **Update SASL constants:** Changes can be seen [here](https://github.com/mostafa/xk6-kafka/pull/86/files#diff-1f48700829d7e0cdba7f3729d6a7c1c5df3aa11f2aeeed583c7e1e39aa7b66d1R18-R22) and are reflected in the [`test_sasl_auth.js`](https://github.com/mostafa/xk6-kafka/pull/86/files#diff-4fc4b76f83b9f6370a1d49020e4e0da57748d6a9a83f1541331d3615d68de581R29-R39) script. Eventually, all of them will be exported at the module level: #70. - **Separated SASL and TLS configs:** An example is available [here](https://github.com/mostafa/xk6-kafka/pull/86/files#diff-4fc4b76f83b9f6370a1d49020e4e0da57748d6a9a83f1541331d3615d68de581R28-R54). - **Fix a bug on multiple occasions:** Almost all the scripts contained [a bug](https://github.com/mostafa/xk6-kafka/pull/86/files#diff-4fc4b76f83b9f6370a1d49020e4e0da57748d6a9a83f1541331d3615d68de581R136) that prevented the correct message to be printed after the topic is deleted, which is now fixed. - **Proper JS API documentation:** 📖 Since the API is changing rapidly, I introduced the `index.d.ts` file and the mechanism to generate JS API docs in the `docs` directory in #87. - **Test against Confluent Cloud and kafka-docker-playground:** I tried [Confluent Cloud](https://confluent.cloud/) with SASL Plain and TLS v1.2 and it worked perfectly and SASL Plain (with no TLS) with [kafka-docker-playgrounds](https://github.com/vdesabou/kafka-docker-playground)'s [sasl-plain](https://github.com/vdesabou/kafka-docker-playground/tree/master/environment/sasl-plain) environment. @fdahunsibread @thanapat-sk @anjosanap @Momotoculteur @kusumkappdirect Please test this PR, so I can make sure it works for you as well. I'd be happy to have your review as well if you're into Go.

volfagnt · June 20, 2022, 8:59am

Hi. I find fix and build version and i have new error:
level=error msg=“x509: certificate relies on legacy Common Name field, use SANs instead”
What’s wrong? In my case common name is required field.
I take certificates from vault in pem format.

mostafa · June 20, 2022, 9:09am

@volfagnt The error is probably a security issue, and you have two options, as stated here:

Re-sign your certificate and add subjectAltName.
Temporarily disable it when building xk6-kafka with xk6 by setting this environment variable GODEBUG="x509ignoreCN=0":

git clone git@github.com:mostafa/xk6-kafka.git && cd xk6-kafka
GODEBUG="x509ignoreCN=0" xk6 build --with github.com/mostafa/xk6-kafka@latest=.

Please let me know if it worked or not.

volfagnt · June 20, 2022, 10:15am

I’m using your add-on as a local folder. And when I try to build I get an error:
./xk6 build GODEBUG=“x509ignoreCN=0” --with github.com/mostafa/xk6-kafka=“E:/some_po/k6-v0.38.3-windows/k6kafka/k6mus/main/xk6-kafka”
2022/06/20 13:09:22 [INFO] Temporary folder: C:\Users\volfg\AppData\Local\Temp\buildenv_2022-06-20-1309.426519271
2022/06/20 13:09:22 [INFO] Initializing Go module
2022/06/20 13:09:22 [INFO] exec (timeout=10s): C:\Program Files\Go\bin\go.exe mod init k6
go: creating new go.mod: module k6
2022/06/20 13:09:23 [INFO] Replace GitHub - mostafa/xk6-kafka: k6 extension to load test Apache Kafka with support for various serialization formats, SASL, TLS, compression, Schema Registry client and beyond => E:/some_po/k6-v0.38.3-windows/k6kafka/k6mus/main/xk6-kafka
2022/06/20 13:09:23 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod edit -replace github.com/mostafa/xk6-kafka=E:/some_po/k6-v0.38.3-windows/k6kafka/k6mus/main/xk6-kafka
2022/06/20 13:09:23 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod tidy -compat=1.17
go: warning: “all” matched no packages
2022/06/20 13:09:23 [INFO] Pinning versions
2022/06/20 13:09:23 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod tidy -compat=1.17
go: found GitHub - mostafa/xk6-kafka: k6 extension to load test Apache Kafka with support for various serialization formats, SASL, TLS, compression, Schema Registry client and beyond in GitHub - mostafa/xk6-kafka: k6 extension to load test Apache Kafka with support for various serialization formats, SASL, TLS, compression, Schema Registry client and beyond v0.0.0-00010101000000-000000000000
go: finding module for package GitHub - fsnotify/fsnotify: Cross-platform file system notifications for Go.
go: finding module for package gopkg.in/tomb.v1
go: found gopkg.in/tomb.v1 in gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: found GitHub - fsnotify/fsnotify: Cross-platform file system notifications for Go. in GitHub - fsnotify/fsnotify: Cross-platform file system notifications for Go. v1.5.4
2022/06/20 13:09:24 [INFO] Writing main module: C:\Users\volfg\AppData\Local\Temp\buildenv_2022-06-20-1309.426519271\main.go
2022/06/20 13:09:24 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod edit -require go.k6.io/k6@GODEBUG=x509ignoreCN=0
2022/06/20 13:09:24 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod tidy -compat=1.17
go: errors parsing go.mod:
C:\Users\volfg\AppData\Local\Temp\buildenv_2022-06-20-1309.426519271\go.mod:31:2: require go.k6.io/k6: version “GODEBUG=x509ignoreCN=0” invalid: unknown revision GODEBUG=x509ignoreCN=0
2022/06/20 13:09:26 [INFO] Cleaning up temporary folder: C:\Users\volfg\AppData\Local\Temp\buildenv_2022-06-20-1309.426519271
2022/06/20 13:09:26 [FATAL] exit status 1
When I use your example I get:

GODEBUG=“x509ignoreCN=0” xk6 build --with github.com/mostafa/xk6-kafk …

  + CategoryInfo          : ObjectNotFound: (GODEBUG=x509ignoreCN=0:String) [], CommandNotFoundException
  + FullyQualifiedErrorId : CommandNotFoundException

mostafa · June 20, 2022, 11:17am

@volfagnt
You are making a mistake by passing the environment variable as a command. Look at the examples here. This is an excellent resource for learning about global variables, a.k.a. environment variables, in BASH, if you are running xk6 on a GNU/Linux machine.

Update:
I just noticed you are running xk6 in PowerShell, which means that you need to read this, that is, you either set the environment variable in the System Properties -> Advanced -> Environment Variables -> System variables (and restarting your shell) or you can pass them directly to pwsh.exe while invoking it like this:

pwsh -Command { $env:GODEBUG="x509ignoreCN=0"; .\xk6 build --with github.com/mostafa/xk6-kafka@latest=. }

volfagnt · June 20, 2022, 10:34pm

Thx, so build new one k6 like that - $Env:GODEBUG =‘x509ignoreCN=0’; ./xk6 build v0.38.3 --with github.com/mostafa/xk6-kafka@v0.11.0
I have created a System variable

And I am passing the certificates as a string, but still get errors(((

My conf

mostafa · June 21, 2022, 9:44am

@volfagnt The certificates should be passed as filenames, not strings. Please have a look at this unit test:

github.com

mostafa/xk6-kafka/blob/main/auth_test.go#L120-L131


      
          	assert.Nil(t, fileExists("auth_test.go"))
          	assert.NotNil(t, fileExists("test.go.not"))
          }
          
          type SimpleTLSConfig struct {
          	tlsConfig TLSConfig
          	err       *Xk6KafkaError
          }
          
          // TestTlsConfig tests the creation of a TLS config.
          func TestTlsConfig(t *testing.T) {
          	tlsConfig := TLSConfig{

volfagnt · June 21, 2022, 10:32am

Hi! All three certs can be .pem??

mostafa · June 21, 2022, 11:01am

@volfagnt According to the following, they must be PEM-encoded:

mostafa · June 21, 2022, 11:03am

@volfagnt Keep an eye on this issue and possibly upvote it, so I know if you want to have JKS instead of PEM because I know what a hassle it is to convert them.

github.com/mostafa/xk6-kafka

Add support for loading JKS files

opened 05:25PM - 05 Jun 22 UTC

closed 03:14PM - 23 Mar 23 UTC

mostafa

✨ Feature Request

For configuring Kafka with SSL, we need JKS files. Still, private keys and certi…ficates in PEM format are currently supported, which means the user needs to extract the keys and certificates manually from the JKS files using the `keytool` application and then convert them to be able to use them with the extension. This feature helps alleviate all these steps. Based on my initial investigation, the [keystore-go](https://github.com/pavlo-v-chernykh/keystore-go) can be a viable candidate for quickly adding the feature. Note to self: - [x] Answer [this person on SO](https://stackoverflow.com/questions/72261550/how-to-work-with-jks-files-in-golang-kafka-go) when you resolved the issue.

volfagnt · June 21, 2022, 12:40pm

I had such a problem on jmeter, if you add bouncycastle lib, then you can bypass jks on spring and java.|
Maybe there is a solution for Go like bouncycastle.

Now for my scenario, I added the ability to create a local file with certificates in .pem from response. But again began to receive an old problem.

My build was like that - $Env:GODEBUG =‘x509ignoreCN=0’; ./xk6 build v0.38.3 --with github.com/mostafa/xk6-kafka@v0.11.0 --with github.com/avitalique/xk6-file@latest
OS Windows Powershell.

mostafa · June 21, 2022, 1:07pm

@volfagnt According to the following link, the feature flag is removed in Go 1.17, so you can no longer pass the env-var to xk6 and Go while compiling xk6-kafka. You must re-sign your certificate and set subjectAltName.

github.com/rancher/rke2

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

opened 06:56AM - 12 Mar 21 UTC

closed 08:47PM - 10 Jun 22 UTC

Martin-Weiss

status/stale

v1.20.4+rke2r1 During deployment we tried to fetch images from the on-premise… registry and there we got the error message `x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0` Root cause seems to be that the certificate used on the registry does NOT have SANs specified and is only using the CN of the host. As the certificate can not be replaced we had to add GODEBUG=x509ignoreCN=0 to /etc/sysconfig/rke2-server as a workaround. I am not sure if this is a defect in GO and why we have to use this workaround. I thought that using CN "only" and "without SANs" is valid and RFC conform.. Do we have to replace all certificates that just have a CN or do we have to get a fixed GO or??