diff --git a/cicd/examples/aws-nginx/README.md b/cicd/examples/aws-nginx/README.md new file mode 100644 index 00000000..b8485620 --- /dev/null +++ b/cicd/examples/aws-nginx/README.md @@ -0,0 +1,148 @@ +# AWS nginx deployment through stackql + +The [resource spec](resources.json), [provider overlay](ec2.merge.json), [stopped-state spec](stopped.json) +and [bootstrap script](user-data.sh) are included here. No Omni checkout or generated SQL is required. +The spec provisions a VPC, two subnets, routing, security group, SSH key pair and nginx VM. + + +Start with **your existing pinned canonical AWS registry** in stackql. The session merge augments +that registry; do not point stackql at Omni's test corpus or its generated effective registry. +Requires stackql's `document_merge` and `inputs_file` support, consuming Omni `v0.1.11-alpha01`. +These stackql steps have not been verified against live AWS. + +Run from the **stackql repository root**. Prepare local SSH keys and nginx bootstrap inputs +(the default AMI lookup is read-only): + +```sh +export CASE="$PWD/.stackql/aws-nginx" +export SSH_CIDR="$(curl --fail --silent --show-error https://checkip.amazonaws.com)/32" +python3 cicd/examples/aws-nginx/prepare.py \ + --case "$CASE" --prefix my-nginx-demo --region ap-southeast-2 \ + --zone-a ap-southeast-2a --zone-b ap-southeast-2b \ + --ssh-cidr "$SSH_CIDR" +``` + +Choose availability zones available to your account. The default image is Amazon Linux 2023 x86_64, +resolved once during preparation through AWS's public SSM parameter. The concrete AMI ID is +persisted in `inputs.json`; launches never use a moving image reference. This default lookup requires +the AWS CLI and `ssm:GetParameter`. `--image-id` with a concrete AMI skips it. Deployment requires +the EC2 permissions. Creates billable AWS resources. HTTP is public; SSH is +restricted to the supplied CIDR. Use `--http-cidr` to restrict HTTP. + +A unique client token is generated once per prepared deployment and persisted alongside the AMI. +Retries/reapply reuse both; preparation refuses to overwrite an existing `inputs.json`. A new case +gets a new token even with the same display prefix. Do not rotate a token while an earlier launch +has an unresolved outcome. + +Open stackql using your usual pinned registry configuration, adding the preview flag: + +```sh +./build/stackql shell --preview '{"unstable":true}' +``` + +Inside that session, merge the overlay and create the deployment: + +```sql +SET "$.document_merge" = '@.stackql/aws-nginx/merge.json'; + +SELECT * FROM stackql_preview.iac.run +WHERE collection = 'my-nginx-demo' + AND region = 'ap-southeast-2' + AND resources_file = 'cicd/examples/aws-nginx/resources.json' + AND inputs_file = '.stackql/aws-nginx/inputs.json'; +``` + +Use the collection and region chosen during preparation. AWS credentials use the normal environment. +Run the same SELECT again to verify unchanged reapply. Each execution gets a fresh run ID automatically. + +## Read managed identities + +```sql +SELECT resource_key, identity, phase +FROM stackql_preview.iac.ledger +WHERE collection = 'my-nginx-demo' AND resource_key = 'instance'; +``` + +Requires an Omni release with the new read-only ledger API (not yet in `v0.1.11-alpha01`). Reads +use the same app/state root as apply, without locks, journals or cloud calls. A pending row's +identity is the previous committed identity, not confirmation of the pending operation. +Explicit `Name` tags are display labels only; ownership and recovery come from the ledger. + +## Query the IP and connect over SSH + +Set the SQL variable `instance_id` to the `identity` returned for the collection's `/instance` row +(`SET instance_id = 'the actual returned identity';`). Then, in the same merged session: + +```sql +SELECT id, public_ip, + CONCAT('ssh -i .stackql/aws-nginx/id_ed25519 ec2-user@', public_ip) AS ssh_command +FROM stackql_unstable_aws.ec2.web_instance +WHERE region = 'ap-southeast-2' + AND "InstanceId.1" = $instance_id; +``` + +Run the returned `ssh_command` in your local terminal. Preparation generated the private key at +`.stackql/aws-nginx/id_ed25519`; its public key was imported into EC2. The private key is local, +not fetched from AWS. If preparation used `--public-key`, substitute its corresponding private key +path. The query returns `public_ip` for the HTTP `/healthz` check too. EC2 running does not guarantee +nginx bootstrap has completed. + +## Stop and destroy + +Declare the VM stopped, then destroy the collection in dependency order: + +```sql +SELECT * FROM stackql_preview.iac.run +WHERE collection = 'my-nginx-demo' + AND region = 'ap-southeast-2' + AND resources_file = 'cicd/examples/aws-nginx/stopped.json' + AND inputs_file = '.stackql/aws-nginx/inputs.json'; + +-- Execute only after the stopped-state result succeeds. +SELECT * FROM stackql_preview.iac.destroy +WHERE collection = 'my-nginx-demo' + AND region = 'ap-southeast-2'; + +-- After teardown, restore the session's canonical document view. +UNSET "$.document_merge"; +``` + +Ledger and journals default to `.stackql/iac` under the working directory (or `--approot`). Merge cache +normally defaults to `.stackql/cache/document-views`; preparation's `merge.json` explicitly selects +its case-local `registry-cache`. Keep the same app root, collection and overlay for every run. +Reopening a session requires repeating SET before continuing. Paths refer to the stackql process. + +For an existing deployment, configure `iacStateRoot` to its original ledger directory before +continuing. A different ledger can create duplicates. Inspect failed/outstanding results before +any further mutation. + +## Recover a pending create, then resume + +First establish the original operation's actual outcome. For an independently verified absent +create that is no longer in flight: + +```sql +SELECT resource_key, phase, version FROM stackql_preview.iac.ledger +WHERE collection = 'my-nginx-demo' AND resource_key = 'subnet_a'; + +SELECT * FROM stackql_preview.iac.recover +WHERE collection = 'my-nginx-demo' + AND resource_key = 'subnet_a' + AND outcome = 'absent' + AND version = '1'; +``` + +Use the version returned by the first query; `1` is only the first-create example. Then repeat the +original apply SELECT with the same collection, state, inputs and resource spec. Existing resources +are retained. Recovery updates local ledger state only, records the operator assertion durably and +checks the pending version under an exclusive lock. It refuses prior committed state or dependents +for absent recovery. If the create completed, use `outcome = 'applied'` and the verified `identity`; +verify completion/readiness, not merely existence. Tags are not ownership or recovery evidence. +Requires an Omni release after `v0.1.11-alpha03` containing recovery. Older releases return an explicit +upgrade error. There is no automatic replay of an uncertain mutation. + +## Offline preparation checks + +```sh +python3 cicd/examples/aws-nginx/test_prepare.py +``` diff --git a/cicd/examples/aws-nginx/ec2.merge.json b/cicd/examples/aws-nginx/ec2.merge.json new file mode 100644 index 00000000..ca75e390 --- /dev/null +++ b/cicd/examples/aws-nginx/ec2.merge.json @@ -0,0 +1,2084 @@ +{ + "components": { + "schemas": { + "WebHandle": { + "properties": { + "id": { + "type": "string" + }, + "ok": { + "type": "boolean" + } + }, + "type": "object" + }, + "WebObservation": { + "properties": { + "id": { + "type": "string" + }, + "public_ip": { + "type": "string" + }, + "state": { + "type": "string" + }, + "State": { + "type": "string" + } + }, + "type": "object" + } + }, + "x-stackQL-resources": { + "web_association_a": { + "id": "aws.ec2.web_association_a", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=AssociateRouteTable&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"AssociateRouteTableResponse\" \"associationId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DisassociateRouteTable&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeRouteTables&Version=2016-11-15&RouteTableId.1={RouteTableId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "items", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ define \"association\" }}{\"id\":{{ toJson (index . \"routeTableAssociationId\") }}}{{ end }}{\"items\":[{{ $r := index . \"DescribeRouteTablesResponse\" }}{{ if index $r \"routeTableSet\" }}{{ $v := index $r \"routeTableSet\" \"item\" }}{{ if index $v \"associationSet\" }}{{ $a := index $v \"associationSet\" \"item\" }}{{ if eq (kindOf $a) \"slice\" }}{{ $sep := separator \",\" }}{{ range $a }}{{ call $sep }}{{ template \"association\" . }}{{ end }}{{ else if eq (kindOf $a) \"map\" }}{{ template \"association\" $a }}{{ end }}{{ end }}{{ end }}]}" + } + } + } + }, + "name": "web_association_a", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_association_a/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_association_a/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_association_a/methods/read_self" + } + ] + } + }, + "web_association_b": { + "id": "aws.ec2.web_association_b", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=AssociateRouteTable&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"AssociateRouteTableResponse\" \"associationId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DisassociateRouteTable&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeRouteTables&Version=2016-11-15&RouteTableId.1={RouteTableId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "items", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ define \"association\" }}{\"id\":{{ toJson (index . \"routeTableAssociationId\") }}}{{ end }}{\"items\":[{{ $r := index . \"DescribeRouteTablesResponse\" }}{{ if index $r \"routeTableSet\" }}{{ $v := index $r \"routeTableSet\" \"item\" }}{{ if index $v \"associationSet\" }}{{ $a := index $v \"associationSet\" \"item\" }}{{ if eq (kindOf $a) \"slice\" }}{{ $sep := separator \",\" }}{{ range $a }}{{ call $sep }}{{ template \"association\" . }}{{ end }}{{ else if eq (kindOf $a) \"map\" }}{{ template \"association\" $a }}{{ end }}{{ end }}{{ end }}]}" + } + } + } + }, + "name": "web_association_b", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_association_b/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_association_b/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_association_b/methods/read_self" + } + ] + } + }, + "web_default_route": { + "id": "aws.ec2.web_default_route", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=CreateRoute&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DeleteRoute&Version=2016-11-15/get" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeRouteTables&Version=2016-11-15&RouteTableId.1={RouteTableId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeRouteTablesResponse\" }}{{ if index $r \"routeTableSet\" }}{{ $v := index $r \"routeTableSet\" \"item\" }}{{ $found := false }}{{ if index $v \"routeSet\" }}{{ $routes := index $v \"routeSet\" \"item\" }}{{ if eq (kindOf $routes) \"slice\" }}{{ range $routes }}{{ if eq (index . \"destinationCidrBlock\") \"0.0.0.0/0\" }}{{ $found = true }}{{ end }}{{ end }}{{ else if eq (kindOf $routes) \"map\" }}{{ if eq (index $routes \"destinationCidrBlock\") \"0.0.0.0/0\" }}{{ $found = true }}{{ end }}{{ end }}{{ end }}{{ if $found }}{\"id\":{{ toJson (index $v \"routeTableId\") }}}{{ else }}{}{{ end }}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_default_route", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_default_route/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_default_route/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_default_route/methods/read_self" + } + ] + } + }, + "web_gateway": { + "id": "aws.ec2.web_gateway", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=CreateInternetGateway&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"CreateInternetGatewayResponse\" \"internetGateway\" \"internetGatewayId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DeleteInternetGateway&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeInternetGateways&Version=2016-11-15&InternetGatewayId.1={InternetGatewayId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeInternetGatewaysResponse\" }}{{ if index $r \"internetGatewaySet\" }}{{ $v := index $r \"internetGatewaySet\" \"item\" }}{\"id\":{{ toJson (index $v \"internetGatewayId\") }}}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_gateway", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_gateway/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_gateway/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_gateway/methods/read_self" + } + ] + } + }, + "web_gateway_attachment": { + "id": "aws.ec2.web_gateway_attachment", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=AttachInternetGateway&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DetachInternetGateway&Version=2016-11-15/get" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeInternetGateways&Version=2016-11-15&InternetGatewayId.1={InternetGatewayId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeInternetGatewaysResponse\" }}{{ if index $r \"internetGatewaySet\" }}{{ $v := index $r \"internetGatewaySet\" \"item\" }}{{ if index $v \"attachmentSet\" }}{\"id\":{{ toJson (index $v \"internetGatewayId\") }}}{{ else }}{}{{ end }}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_gateway_attachment", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_gateway_attachment/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_gateway_attachment/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_gateway_attachment/methods/read_self" + } + ] + } + }, + "web_ingress": { + "id": "aws.ec2.web_ingress", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=AuthorizeSecurityGroupIngress&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=RevokeSecurityGroupIngress&Version=2016-11-15/get" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeSecurityGroups&Version=2016-11-15&GroupId.1={GroupId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeSecurityGroupsResponse\" }}{{ if index $r \"securityGroupInfo\" }}{{ $v := index $r \"securityGroupInfo\" \"item\" }}{{ if index $v \"ipPermissions\" }}{\"id\":{{ toJson (index $v \"groupId\") }}}{{ else }}{}{{ end }}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_ingress", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_ingress/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_ingress/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_ingress/methods/read_self" + } + ] + } + }, + "web_instance": { + "id": "aws.ec2.web_instance", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=RunInstances&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"RunInstancesResponse\" \"instancesSet\" \"item\" \"instanceId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + }, + "x-stackQL-beta": { + "await": { + "done": "running", + "interval": "2s", + "maxAttempts": 120, + "statusPath": "state" + }, + "bind": { + "InstanceId.1": "id" + }, + "method": { + "$ref": "#/components/x-stackQL-resources/web_instance/methods/wait_running" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=TerminateInstances&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"TerminateInstancesResponse\" \"instancesSet\" \"item\" \"instanceId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + }, + "x-stackQL-beta": { + "await": { + "done": "terminated", + "interval": "2s", + "maxAttempts": 120, + "statusPath": "state" + }, + "bind": { + "InstanceId.1": "id" + }, + "method": { + "$ref": "#/components/x-stackQL-resources/web_instance/methods/wait_terminated" + } + } + }, + "stop": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=StopInstances&Version=2016-11-15&InstanceId.1={InstanceId.1}/get" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"StopInstancesResponse\" \"instancesSet\" \"item\" \"instanceId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + }, + "x-stackQL-beta": { + "await": { + "done": "stopped", + "interval": "2s", + "maxAttempts": 120, + "statusPath": "state" + }, + "bind": { + "InstanceId.1": "id" + }, + "method": { + "$ref": "#/components/x-stackQL-resources/web_instance/methods/wait_stopped" + } + } + }, + "wait_running": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DescribeInstances&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "transform": { + "body": "{{ $r := index . \"DescribeInstancesResponse\" }}{{ if index $r \"reservationSet\" }}{{ $v := index $r \"reservationSet\" \"item\" \"instancesSet\" \"item\" }}{\"id\":{{ toJson (index $v \"instanceId\") }},\"state\":{{ toJson (index $v \"instanceState\" \"name\") }},\"public_ip\":{{ toJson (index $v \"ipAddress\") }}}{{ else }}{\"state\":\"unknown\"}{{ end }}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "wait_stopped": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DescribeInstances&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "transform": { + "body": "{{ $r := index . \"DescribeInstancesResponse\" }}{{ if index $r \"reservationSet\" }}{{ $v := index $r \"reservationSet\" \"item\" \"instancesSet\" \"item\" }}{\"id\":{{ toJson (index $v \"instanceId\") }},\"state\":{{ toJson (index $v \"instanceState\" \"name\") }},\"public_ip\":{{ toJson (index $v \"ipAddress\") }}}{{ else }}{\"state\":\"unknown\"}{{ end }}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "wait_terminated": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DescribeInstances&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "transform": { + "body": "{{ $r := index . \"DescribeInstancesResponse\" }}{{ if index $r \"reservationSet\" }}{{ $v := index $r \"reservationSet\" \"item\" \"instancesSet\" \"item\" }}{\"id\":{{ toJson (index $v \"instanceId\") }},\"state\":{{ toJson (index $v \"instanceState\" \"name\") }},\"public_ip\":{{ toJson (index $v \"ipAddress\") }}}{{ else }}{\"state\":\"unknown\"}{{ end }}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeInstances&Version=2016-11-15&InstanceId.1={InstanceId.1}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeInstancesResponse\" }}{{ if index $r \"reservationSet\" }}{{ $v := index $r \"reservationSet\" \"item\" }}{{ $v = index $v \"instancesSet\" \"item\" }}{\"id\":{{ toJson (index $v \"instanceId\") }},\"State\":{{ toJson (index $v \"instanceState\" \"name\") }},\"public_ip\":{{ toJson (index $v \"ipAddress\") }}}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_instance", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_instance/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_instance/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_instance/methods/read_self" + } + ], + "update": [ + { + "$ref": "#/components/x-stackQL-resources/web_instance/methods/stop" + } + ] + }, + "updates": [ + { + "priority": 100, + "requires": [ + "State" + ], + "permits": [ + "State" + ], + "target": { + "$ref": "#/components/x-stackQL-resources/web_instance/methods/stop" + } + } + ] + }, + "web_key": { + "id": "aws.ec2.web_key", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=ImportKeyPair&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"ImportKeyPairResponse\" \"keyName\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DeleteKeyPair&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeKeyPairs&Version=2016-11-15&KeyName.1={KeyName}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeKeyPairsResponse\" }}{{ if index $r \"keySet\" }}{{ $v := index $r \"keySet\" \"item\" }}{\"id\":{{ toJson (index $v \"keyName\") }}}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_key", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_key/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_key/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_key/methods/read_self" + } + ] + } + }, + "web_route_table": { + "id": "aws.ec2.web_route_table", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=CreateRouteTable&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"CreateRouteTableResponse\" \"routeTable\" \"routeTableId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DeleteRouteTable&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeRouteTables&Version=2016-11-15&RouteTableId.1={RouteTableId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeRouteTablesResponse\" }}{{ if index $r \"routeTableSet\" }}{{ $v := index $r \"routeTableSet\" \"item\" }}{\"id\":{{ toJson (index $v \"routeTableId\") }}}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_route_table", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_route_table/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_route_table/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_route_table/methods/read_self" + } + ] + } + }, + "web_security_group": { + "id": "aws.ec2.web_security_group", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=CreateSecurityGroup&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"CreateSecurityGroupResponse\" \"groupId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DeleteSecurityGroup&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeSecurityGroups&Version=2016-11-15&GroupId.1={GroupId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeSecurityGroupsResponse\" }}{{ if index $r \"securityGroupInfo\" }}{{ $v := index $r \"securityGroupInfo\" \"item\" }}{\"id\":{{ toJson (index $v \"groupId\") }}}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_security_group", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_security_group/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_security_group/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_security_group/methods/read_self" + } + ] + } + }, + "web_subnet_a": { + "id": "aws.ec2.web_subnet_a", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=CreateSubnet&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"CreateSubnetResponse\" \"subnet\" \"subnetId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + }, + "x-stackQL-beta": { + "await": { + "done": "available", + "interval": "2s", + "maxAttempts": 120, + "statusPath": "state" + }, + "bind": { + "SubnetId.1": "id" + }, + "method": { + "$ref": "#/components/x-stackQL-resources/web_subnet_a/methods/wait_available" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DeleteSubnet&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "wait_available": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DescribeSubnets&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "transform": { + "body": "{{ $v := index . \"DescribeSubnetsResponse\" \"subnetSet\" \"item\" }}{\"id\":{{ toJson (index $v \"subnetId\") }},\"state\":{{ toJson (index $v \"state\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeSubnets&Version=2016-11-15&SubnetId.1={SubnetId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeSubnetsResponse\" }}{{ if index $r \"subnetSet\" }}{{ $v := index $r \"subnetSet\" \"item\" }}{\"id\":{{ toJson (index $v \"subnetId\") }}}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_subnet_a", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_subnet_a/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_subnet_a/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_subnet_a/methods/read_self" + } + ] + } + }, + "web_subnet_b": { + "id": "aws.ec2.web_subnet_b", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=CreateSubnet&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"CreateSubnetResponse\" \"subnet\" \"subnetId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + }, + "x-stackQL-beta": { + "await": { + "done": "available", + "interval": "2s", + "maxAttempts": 120, + "statusPath": "state" + }, + "bind": { + "SubnetId.1": "id" + }, + "method": { + "$ref": "#/components/x-stackQL-resources/web_subnet_b/methods/wait_available" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DeleteSubnet&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "wait_available": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DescribeSubnets&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "transform": { + "body": "{{ $v := index . \"DescribeSubnetsResponse\" \"subnetSet\" \"item\" }}{\"id\":{{ toJson (index $v \"subnetId\") }},\"state\":{{ toJson (index $v \"state\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeSubnets&Version=2016-11-15&SubnetId.1={SubnetId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeSubnetsResponse\" }}{{ if index $r \"subnetSet\" }}{{ $v := index $r \"subnetSet\" \"item\" }}{\"id\":{{ toJson (index $v \"subnetId\") }}}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_subnet_b", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_subnet_b/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_subnet_b/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_subnet_b/methods/read_self" + } + ] + } + }, + "web_vpc": { + "id": "aws.ec2.web_vpc", + "methods": { + "create": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=CreateVpc&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"id\":{{ toJson (index . \"CreateVpcResponse\" \"vpc\" \"vpcId\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + }, + "x-stackQL-beta": { + "await": { + "done": "available", + "interval": "2s", + "maxAttempts": 120, + "statusPath": "state" + }, + "bind": { + "VpcId.1": "id" + }, + "method": { + "$ref": "#/components/x-stackQL-resources/web_vpc/methods/wait_available" + } + } + }, + "delete": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DeleteVpc&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebHandle" + }, + "transform": { + "body": "{\"ok\":true}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "wait_available": { + "config": { + "queryParamTranspose": { + "algorithm": "AWSCanonical" + }, + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + } + }, + "operation": { + "$ref": "#/paths/~1?Action=DescribeVpcs&Version=2016-11-15/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "response": { + "mediaType": "application/xml", + "objectKey": "$", + "openAPIDocKey": "200", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "transform": { + "body": "{{ $v := index . \"DescribeVpcsResponse\" \"vpcSet\" \"item\" }}{\"id\":{{ toJson (index $v \"vpcId\") }},\"state\":{{ toJson (index $v \"state\") }}}", + "type": "golang_template_mxj_v0.1.0" + } + } + }, + "read_self": { + "operation": { + "$ref": "#/paths/~1?Action=DescribeVpcs&Version=2016-11-15&VpcId.1={VpcId}/get" + }, + "request": { + "nativeCasing": "pascal" + }, + "config": { + "requestTranslate": { + "algorithm": "get_query_to_post_form_utf_8" + }, + "queryParamTranspose": { + "algorithm": "AWSCanonical" + } + }, + "response": { + "openAPIDocKey": "200", + "mediaType": "application/xml", + "overrideMediaType": "application/json", + "schema_override": { + "$ref": "#/components/schemas/WebObservation" + }, + "objectKey": "$", + "transform": { + "type": "golang_template_mxj_v0.1.0", + "body": "{{ $r := index . \"DescribeVpcsResponse\" }}{{ if index $r \"vpcSet\" }}{{ $v := index $r \"vpcSet\" \"item\" }}{\"id\":{{ toJson (index $v \"vpcId\") }}}{{ else }}{}{{ end }}" + } + } + } + }, + "name": "web_vpc", + "sqlVerbs": { + "delete": [ + { + "$ref": "#/components/x-stackQL-resources/web_vpc/methods/delete" + } + ], + "insert": [ + { + "$ref": "#/components/x-stackQL-resources/web_vpc/methods/create" + } + ], + "select": [ + { + "$ref": "#/components/x-stackQL-resources/web_vpc/methods/read_self" + } + ] + } + } + } + }, + "paths": { + "/?Action=StopInstances&Version=2016-11-15": { + "get": { + "parameters": [ + { + "description": "The IDs of the instances.", + "in": "query", + "name": "InstanceId", + "required": false, + "schema": { + "$ref": "#/components/schemas/InstanceIdStringList" + } + }, + { + "description": "Hibernates the instance if the instance was enabled for hibernation at launch. If the instance cannot hibernate successfully, a normal shutdown occurs. For more information, see Hibernate your Amazon EC2 instance in the Amazon EC2 User Guide. Default: false", + "in": "query", + "name": "Hibernate", + "required": false, + "schema": { + "$ref": "#/components/schemas/Boolean" + } + }, + { + "description": "Specifies whether to bypass the graceful OS shutdown process when the instance is stopped. Bypassing the graceful OS shutdown might result in data loss or corruption (for example, memory contents not flushed to disk or loss of in-flight IOs) or skipped shutdown scripts. Default: false", + "in": "query", + "name": "SkipOsShutdown", + "required": false, + "schema": { + "$ref": "#/components/schemas/Boolean" + } + }, + { + "description": "Checks whether you have the required permissions for the operation, without actually making the request, and provides an error response. If you have the required permissions, the error response is DryRunOperation. Otherwise, it is UnauthorizedOperation.", + "in": "query", + "name": "DryRun", + "required": false, + "schema": { + "$ref": "#/components/schemas/Boolean" + } + }, + { + "description": "Forces the instance to stop. The instance will first attempt a graceful shutdown, which includes flushing file system caches and metadata. If the graceful shutdown fails to complete within the timeout period, the instance shuts down forcibly without flushing the file system caches and metadata. After using this option, you must perform file system check and repair procedures. This option is not recommended for Windows instances. For more information, see Troubleshoot Amazon EC2 instance stop issues in the Amazon EC2 User Guide. Default: false", + "in": "query", + "name": "Force", + "required": false, + "schema": { + "$ref": "#/components/schemas/Boolean" + } + }, + { + "in": "query", + "name": "InstanceId.1", + "required": true, + "schema": { + "type": "string" + } + } + ] + } + }, + "/?Action=TerminateInstances&Version=2016-11-15": { + "get": { + "parameters": [ + { + "description": "The IDs of the instances. Constraints: Up to 1000 instance IDs. We recommend breaking up this request into smaller batches.", + "in": "query", + "name": "InstanceId", + "required": false, + "schema": { + "$ref": "#/components/schemas/InstanceIdStringList" + } + }, + { + "description": "Forces the instances to terminate. The instance will first attempt a graceful shutdown, which includes flushing file system caches and metadata. If the graceful shutdown fails to complete within the timeout period, the instance shuts down forcibly without flushing the file system caches and metadata.", + "in": "query", + "name": "Force", + "required": false, + "schema": { + "$ref": "#/components/schemas/Boolean" + } + }, + { + "description": "Specifies whether to bypass the graceful OS shutdown process when the instance is terminated. Default: false", + "in": "query", + "name": "SkipOsShutdown", + "required": false, + "schema": { + "$ref": "#/components/schemas/Boolean" + } + }, + { + "description": "Checks whether you have the required permissions for the operation, without actually making the request, and provides an error response. If you have the required permissions, the error response is DryRunOperation. Otherwise, it is UnauthorizedOperation.", + "in": "query", + "name": "DryRun", + "required": false, + "schema": { + "$ref": "#/components/schemas/Boolean" + } + }, + { + "in": "query", + "name": "InstanceId.1", + "required": true, + "schema": { + "type": "string" + } + } + ] + } + }, + "/?Action=DescribeVpcs&Version=2016-11-15&VpcId.1={VpcId}": { + "get": { + "operationId": "WebRead_vpc", + "parameters": [ + { + "name": "VpcId", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Read the addressed object" + } + } + } + }, + "/?Action=DescribeSubnets&Version=2016-11-15&SubnetId.1={SubnetId}": { + "get": { + "operationId": "WebRead_subnet_b", + "parameters": [ + { + "name": "SubnetId", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Read the addressed object" + } + } + } + }, + "/?Action=DescribeInternetGateways&Version=2016-11-15&InternetGatewayId.1={InternetGatewayId}": { + "get": { + "operationId": "WebRead_gateway_attachment", + "parameters": [ + { + "name": "InternetGatewayId", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Read the addressed object" + } + } + } + }, + "/?Action=DescribeRouteTables&Version=2016-11-15&RouteTableId.1={RouteTableId}": { + "get": { + "operationId": "WebListAssociations", + "parameters": [ + { + "name": "RouteTableId", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Table associations" + } + } + } + }, + "/?Action=DescribeRouteTables&Version=2016-11-15&Filter.1.Value.1={AssociationId}&Filter.1.Name=association.association-id": { + "get": { + "operationId": "WebRead_association_b", + "parameters": [ + { + "name": "AssociationId", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Read the addressed object" + } + } + } + }, + "/?Action=DescribeSecurityGroups&Version=2016-11-15&GroupId.1={GroupId}": { + "get": { + "operationId": "WebRead_ingress", + "parameters": [ + { + "name": "GroupId", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Read the addressed object" + } + } + } + }, + "/?Action=DescribeKeyPairs&Version=2016-11-15&KeyName.1={KeyName}": { + "get": { + "operationId": "WebRead_key", + "parameters": [ + { + "name": "KeyName", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Read the addressed object" + } + } + } + }, + "/?Action=DescribeInstances&Version=2016-11-15&InstanceId.1={InstanceId.1}": { + "get": { + "operationId": "WebRead_instance", + "parameters": [ + { + "name": "InstanceId.1", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "InstanceId", + "in": "query", + "required": false, + "schema": { + "type": "array", + "items": { + "type": "string" + } + } + } + ], + "responses": { + "200": { + "description": "Read the addressed object" + } + } + } + }, + "/?Action=StopInstances&Version=2016-11-15&InstanceId.1={InstanceId.1}": { + "get": { + "operationId": "WebStopInstance", + "parameters": [ + { + "name": "InstanceId.1", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "State", + "in": "path", + "required": true, + "schema": { + "type": "string", + "enum": [ + "stopped" + ] + } + }, + { + "name": "InstanceId", + "in": "query", + "required": false, + "schema": { + "type": "array", + "items": { + "type": "string" + } + } + } + ], + "responses": { + "200": { + "description": "Stop requested" + } + } + } + } + } +} diff --git a/cicd/examples/aws-nginx/image_inputs.py b/cicd/examples/aws-nginx/image_inputs.py new file mode 100644 index 00000000..5756944e --- /dev/null +++ b/cicd/examples/aws-nginx/image_inputs.py @@ -0,0 +1,20 @@ +"""Resolve the image once during preparation, never during a launch retry.""" +import re +import subprocess + +PARAMETER = '/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64' + + +def resolve_image(image_id, region, runner=subprocess.run): + if image_id is None: + try: + result = runner([ + 'aws', 'ssm', 'get-parameter', '--region', region, + '--name', PARAMETER, '--query', 'Parameter.Value', '--output', 'text', + ], check=True, capture_output=True, text=True, timeout=30) + except subprocess.CalledProcessError as error: + raise ValueError('AMI lookup failed: ' + (error.stderr or str(error)).strip()) from error + image_id = result.stdout.strip() + if not re.fullmatch(r'ami-(?:[0-9a-f]{8}|[0-9a-f]{17})', image_id): + raise ValueError('image must be a concrete AMI ID; moving resolve:ssm references are not accepted') + return image_id diff --git a/cicd/examples/aws-nginx/prepare.py b/cicd/examples/aws-nginx/prepare.py new file mode 100644 index 00000000..caa2b88a --- /dev/null +++ b/cicd/examples/aws-nginx/prepare.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +"""Prepare local inputs and a document merge request. Resolves an AMI read-only; never provisions cloud resources.""" +import argparse +import base64 +import json +import ipaddress +from pathlib import Path +import re +import subprocess +import uuid +from image_inputs import resolve_image + +p=argparse.ArgumentParser(description=__doc__) +p.add_argument('--case',type=Path,required=True) +p.add_argument('--region', required=True) +p.add_argument('--image-id', help='Concrete AMI ID; otherwise resolve AL2023 through AWS SSM once and persist it') +p.add_argument('--zone-a',required=True) +p.add_argument('--zone-b',required=True) +p.add_argument('--ssh-cidr',required=True) +p.add_argument('--http-cidr',default='0.0.0.0/0') +p.add_argument('--prefix',default='omni-web') +p.add_argument('--instance-type',default='t3.micro') +p.add_argument('--public-key',type=Path) +p.add_argument('--poll-interval',default='2s') +p.add_argument('--poll-attempts',type=int,default=120) +a=p.parse_args() +if not re.fullmatch(r'[A-Za-z0-9-]{1,40}',a.prefix):p.error('invalid prefix') +for cidr in (a.ssh_cidr,a.http_cidr): + if ipaddress.ip_network(cidr).version!=4:p.error('IPv4 CIDRs required') +if a.poll_attempts<1:p.error('poll attempts must be positive') +a.case=a.case.resolve();a.case.mkdir(mode=0o700,parents=True,exist_ok=True) +if (a.case/'inputs.json').exists():p.error('case already prepared; use its inputs and ledger, or a new directory') +try: + a.image_id = resolve_image(a.image_id, a.region) +except (ValueError, OSError, subprocess.SubprocessError) as error: + p.error('cannot pin AMI: ' + str(error)) + +key=a.public_key +if key is None: + private=a.case/'id_ed25519' + if private.exists():p.error('refusing to overwrite local SSH key') + subprocess.run(['ssh-keygen','-q','-t','ed25519','-N','','-C',a.prefix,'-f',str(private)],check=True) + key=private.with_suffix('.pub') +public=key.read_text().strip() +if not public.startswith('ssh-'):p.error('OpenSSH public key required') +base=Path(__file__).resolve().parent +script=(base/'user-data.sh').read_text().replace('__DEPLOYMENT__',a.prefix) +inputs={'image_id':a.image_id,'instance_type':a.instance_type,'zone_a':a.zone_a,'zone_b':a.zone_b,'ssh_cidr':a.ssh_cidr,'http_cidr':a.http_cidr,'key_name':a.prefix+'-ssh','group_name':a.prefix+'-web','client_token':uuid.uuid4().hex,'public_key_material':base64.b64encode(public.encode()).decode(),'user_data':base64.b64encode(script.encode()).decode()} +# Tags are user inputs to the static deployment specification. +for resource in json.loads((base/'resources.json').read_text())['resources']: + inputs['tag_'+resource['key']]=a.prefix+'-'+resource['key'] +patch=json.loads((base/'ec2.merge.json').read_text()) +for resource in patch['components']['x-stackQL-resources'].values(): + for method in resource['methods'].values(): + waiter=method.get('x-stackQL-beta',{}).get('await') + if waiter:waiter.update(interval=a.poll_interval,maxAttempts=a.poll_attempts) +for name,data in {'inputs.json':inputs,'merge.json':{'patches':[{'service':'stackql_unstable_aws.ec2','merge':patch}],'doc_cache':{'dir':str(a.case/'registry-cache')}}}.items(): + path=a.case/name;path.write_text(json.dumps(data,indent=2)+'\n');path.chmod(0o600) +(a.case/'user-data.sh').write_text(script) diff --git a/cicd/examples/aws-nginx/resources.json b/cicd/examples/aws-nginx/resources.json new file mode 100644 index 00000000..70b0a98d --- /dev/null +++ b/cicd/examples/aws-nginx/resources.json @@ -0,0 +1,327 @@ +{ + "resources": [ + { + "key": "vpc", + "provider": "aws", + "address": "ec2.web_vpc", + "identity": "id", + "addressed_by": "VpcId", + "write_once": { + "CidrBlock": "10.77.0.0/16", + "TagSpecification.1.ResourceType": "vpc", + "TagSpecification.1.Tag.1.Key": "Name", + "TagSpecification.1.Tag.1.Value": { + "from": "input", + "attribute": "tag_vpc" + } + }, + "desired": {} + }, + { + "key": "subnet_a", + "provider": "aws", + "address": "ec2.web_subnet_a", + "identity": "id", + "addressed_by": "SubnetId", + "write_once": { + "AvailabilityZone": { + "from": "input", + "attribute": "zone_a" + }, + "CidrBlock": "10.77.1.0/24", + "TagSpecification.1.ResourceType": "subnet", + "TagSpecification.1.Tag.1.Key": "Name", + "TagSpecification.1.Tag.1.Value": { + "from": "input", + "attribute": "tag_subnet_a" + } + }, + "desired": {}, + "inbound": [ + { + "from": "vpc", + "as": "VpcId" + } + ] + }, + { + "key": "subnet_b", + "provider": "aws", + "address": "ec2.web_subnet_b", + "identity": "id", + "addressed_by": "SubnetId", + "write_once": { + "AvailabilityZone": { + "from": "input", + "attribute": "zone_b" + }, + "CidrBlock": "10.77.2.0/24", + "TagSpecification.1.ResourceType": "subnet", + "TagSpecification.1.Tag.1.Key": "Name", + "TagSpecification.1.Tag.1.Value": { + "from": "input", + "attribute": "tag_subnet_b" + } + }, + "desired": {}, + "inbound": [ + { + "from": "vpc", + "as": "VpcId" + } + ] + }, + { + "key": "gateway", + "provider": "aws", + "address": "ec2.web_gateway", + "identity": "id", + "addressed_by": "InternetGatewayId", + "write_once": { + "TagSpecification.1.ResourceType": "internet-gateway", + "TagSpecification.1.Tag.1.Key": "Name", + "TagSpecification.1.Tag.1.Value": { + "from": "input", + "attribute": "tag_gateway" + } + }, + "desired": {} + }, + { + "key": "gateway_attachment", + "provider": "aws", + "address": "ec2.web_gateway_attachment", + "identity": "id", + "addressed_by": "InternetGatewayId", + "write_once": {}, + "desired": {}, + "inbound": [ + { + "from": "gateway", + "as": "InternetGatewayId" + }, + { + "from": "vpc", + "as": "VpcId" + } + ] + }, + { + "key": "route_table", + "provider": "aws", + "address": "ec2.web_route_table", + "identity": "id", + "addressed_by": "RouteTableId", + "write_once": { + "TagSpecification.1.ResourceType": "route-table", + "TagSpecification.1.Tag.1.Key": "Name", + "TagSpecification.1.Tag.1.Value": { + "from": "input", + "attribute": "tag_route_table" + } + }, + "desired": {}, + "inbound": [ + { + "from": "vpc", + "as": "VpcId" + } + ] + }, + { + "key": "default_route", + "provider": "aws", + "address": "ec2.web_default_route", + "identity": "id", + "addressed_by": "RouteTableId", + "write_once": { + "DestinationCidrBlock": "0.0.0.0/0" + }, + "desired": {}, + "inbound": [ + { + "from": "route_table", + "as": "RouteTableId" + }, + { + "from": "gateway_attachment", + "as": "GatewayId" + } + ] + }, + { + "key": "association_a", + "provider": "aws", + "address": "ec2.web_association_a", + "identity": "id", + "addressed_by": "AssociationId", + "write_once": {}, + "desired": {}, + "inbound": [ + { + "from": "route_table", + "as": "RouteTableId" + }, + { + "from": "subnet_a", + "as": "SubnetId" + } + ] + }, + { + "key": "association_b", + "provider": "aws", + "address": "ec2.web_association_b", + "identity": "id", + "addressed_by": "AssociationId", + "write_once": {}, + "desired": {}, + "inbound": [ + { + "from": "route_table", + "as": "RouteTableId" + }, + { + "from": "subnet_b", + "as": "SubnetId" + } + ] + }, + { + "key": "security_group", + "provider": "aws", + "address": "ec2.web_security_group", + "identity": "id", + "addressed_by": "GroupId", + "write_once": { + "GroupDescription": "omnisdk nginx demo", + "GroupName": { + "from": "input", + "attribute": "group_name" + }, + "TagSpecification.1.ResourceType": "security-group", + "TagSpecification.1.Tag.1.Key": "Name", + "TagSpecification.1.Tag.1.Value": { + "from": "input", + "attribute": "tag_security_group" + } + }, + "desired": {}, + "inbound": [ + { + "from": "vpc", + "as": "VpcId" + } + ] + }, + { + "key": "ingress", + "provider": "aws", + "address": "ec2.web_ingress", + "identity": "id", + "addressed_by": "GroupId", + "write_once": { + "IpPermissions.1.FromPort": "22", + "IpPermissions.1.IpProtocol": "tcp", + "IpPermissions.1.IpRanges.1.CidrIp": { + "from": "input", + "attribute": "ssh_cidr" + }, + "IpPermissions.1.ToPort": "22", + "IpPermissions.2.FromPort": "80", + "IpPermissions.2.IpProtocol": "tcp", + "IpPermissions.2.IpRanges.1.CidrIp": { + "from": "input", + "attribute": "http_cidr" + }, + "IpPermissions.2.ToPort": "80" + }, + "desired": {}, + "inbound": [ + { + "from": "security_group", + "as": "GroupId" + } + ] + }, + { + "key": "key", + "provider": "aws", + "address": "ec2.web_key", + "identity": "id", + "addressed_by": "KeyName", + "write_once": { + "KeyName": { + "from": "input", + "attribute": "key_name" + }, + "PublicKeyMaterial": { + "from": "input", + "attribute": "public_key_material" + } + }, + "desired": {} + }, + { + "key": "instance", + "provider": "aws", + "address": "ec2.web_instance", + "identity": "id", + "addressed_by": "InstanceId.1", + "write_once": { + "ClientToken": { + "from": "input", + "attribute": "client_token" + }, + "ImageId": { + "from": "input", + "attribute": "image_id" + }, + "InstanceType": { + "from": "input", + "attribute": "instance_type" + }, + "MaxCount": "1", + "MetadataOptions.HttpTokens": "required", + "MinCount": "1", + "NetworkInterface.1.AssociatePublicIpAddress": "true", + "NetworkInterface.1.DeviceIndex": "0", + "TagSpecification.1.ResourceType": "instance", + "TagSpecification.1.Tag.1.Key": "Name", + "TagSpecification.1.Tag.1.Value": { + "from": "input", + "attribute": "tag_instance" + }, + "UserData": { + "from": "input", + "attribute": "user_data" + } + }, + "desired": {}, + "inbound": [ + { + "from": "subnet_a", + "as": "subnet_id" + }, + { + "from": "key", + "as": "key_name" + }, + { + "from": "ingress", + "as": "security_group" + }, + { + "from": "default_route", + "as": "route_ready" + }, + { + "from": "association_a", + "as": "association_ready" + } + ], + "via_type": "starlark", + "via": "{\"NetworkInterface.1.SubnetId\":inbox[\"subnet_id\"],\"NetworkInterface.1.SecurityGroupId.1\":inbox[\"security_group\"],\"KeyName\":inbox[\"key_name\"]}" + } + ] +} diff --git a/cicd/examples/aws-nginx/stopped.json b/cicd/examples/aws-nginx/stopped.json new file mode 100644 index 00000000..9821166f --- /dev/null +++ b/cicd/examples/aws-nginx/stopped.json @@ -0,0 +1,67 @@ +{ + "resources": [ + { + "key": "instance", + "provider": "aws", + "address": "ec2.web_instance", + "identity": "id", + "addressed_by": "InstanceId.1", + "write_once": { + "ClientToken": { + "from": "input", + "attribute": "client_token" + }, + "ImageId": { + "from": "input", + "attribute": "image_id" + }, + "InstanceType": { + "from": "input", + "attribute": "instance_type" + }, + "MaxCount": "1", + "MetadataOptions.HttpTokens": "required", + "MinCount": "1", + "NetworkInterface.1.AssociatePublicIpAddress": "true", + "NetworkInterface.1.DeviceIndex": "0", + "TagSpecification.1.ResourceType": "instance", + "TagSpecification.1.Tag.1.Key": "Name", + "TagSpecification.1.Tag.1.Value": { + "from": "input", + "attribute": "tag_instance" + }, + "UserData": { + "from": "input", + "attribute": "user_data" + } + }, + "desired": { + "State": "stopped" + }, + "inbound": [ + { + "from": "subnet_a", + "as": "subnet_id" + }, + { + "from": "key", + "as": "key_name" + }, + { + "from": "ingress", + "as": "security_group" + }, + { + "from": "default_route", + "as": "route_ready" + }, + { + "from": "association_a", + "as": "association_ready" + } + ], + "via_type": "starlark", + "via": "{\"NetworkInterface.1.SubnetId\":inbox[\"subnet_id\"],\"NetworkInterface.1.SecurityGroupId.1\":inbox[\"security_group\"],\"KeyName\":inbox[\"key_name\"]}" + } + ] +} diff --git a/cicd/examples/aws-nginx/test_prepare.py b/cicd/examples/aws-nginx/test_prepare.py new file mode 100644 index 00000000..1ea118af --- /dev/null +++ b/cicd/examples/aws-nginx/test_prepare.py @@ -0,0 +1,72 @@ +"""Offline checks of executable image defaults and explicit pinning.""" +import json +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import Mock +from image_inputs import resolve_image, PARAMETER + + +class PrepareInputsTest(unittest.TestCase): + def prepare(self, *extra): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + public = root / 'existing.pub' + public.write_text('ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFixture offline-test\n') + case = root / 'case' + result = subprocess.run([ + sys.executable, str(Path(__file__).with_name('prepare.py')), + '--case', str(case), '--region', 'ap-southeast-2', '--image-id', 'ami-00000000000000001', '--zone-a', 'ap-southeast-2a', '--zone-b', 'ap-southeast-2b', + '--ssh-cidr', '203.0.113.7/32', '--public-key', str(public), *extra, + ], capture_output=True, text=True) + self.assertEqual(result.returncode, 0, result.stderr) + return json.loads((case / 'inputs.json').read_text()) + + def test_prepared_image_is_concrete(self): + self.assertEqual(self.prepare()['image_id'], 'ami-00000000000000001') + + def test_default_resolves_once_before_launch(self): + runner = Mock(return_value=Mock(stdout='ami-00000000000000002\n')) + self.assertEqual(resolve_image(None, 'ap-southeast-2', runner), 'ami-00000000000000002') + runner.assert_called_once() + self.assertIn(PARAMETER, runner.call_args.args[0]) + self.assertIn('ap-southeast-2', runner.call_args.args[0]) + + def test_pinned_image_never_looks_up_latest(self): + runner = Mock() + self.assertEqual(resolve_image('ami-00000000000000001', 'ap-southeast-2', runner), 'ami-00000000000000001') + runner.assert_not_called() + + def test_moving_reference_is_rejected(self): + with self.assertRaises(ValueError): + resolve_image('resolve:ssm:' + PARAMETER, 'ap-southeast-2') + + def test_separate_deployments_do_not_recycle_client_tokens(self): + first, second = self.prepare(), self.prepare() + self.assertNotEqual(first['client_token'], second['client_token']) + self.assertLessEqual(len(first['client_token']), 64) + + def test_existing_inputs_are_not_rotated(self): + with tempfile.TemporaryDirectory() as temporary: + case = Path(temporary) + original = '{"image_id":"ami-00000000000000001","client_token":"persisted-token"}' + (case / 'inputs.json').write_text(original) + result = subprocess.run([ + sys.executable, str(Path(__file__).with_name('prepare.py')), + '--case', str(case), '--region', 'ap-southeast-2', + '--zone-a', 'ap-southeast-2a', '--zone-b', 'ap-southeast-2b', + '--ssh-cidr', '203.0.113.7/32', + ], capture_output=True, text=True) + self.assertNotEqual(result.returncode, 0) + self.assertIn('case already prepared', result.stderr) + self.assertEqual((case / 'inputs.json').read_text(), original) + + def test_explicit_image_can_be_pinned(self): + image = 'ami-00000000000000001' + self.assertEqual(self.prepare('--image-id', image)['image_id'], image) + + +if __name__ == '__main__': + unittest.main() diff --git a/cicd/examples/aws-nginx/user-data.sh b/cicd/examples/aws-nginx/user-data.sh new file mode 100644 index 00000000..e95317a6 --- /dev/null +++ b/cicd/examples/aws-nginx/user-data.sh @@ -0,0 +1,28 @@ +#!/bin/bash +set -euxo pipefail +dnf install -y nginx +cat > /etc/nginx/nginx.conf <<'NGINX' +user nginx; +worker_processes auto; +pid /run/nginx.pid; +error_log /var/log/nginx/error.log; +events { worker_connections 1024; } +http { + include /etc/nginx/mime.types; + default_type application/octet-stream; + access_log /var/log/nginx/access.log; + server { + listen 80 default_server; + server_name _; + root /usr/share/nginx/html; + add_header X-Omnisdk-Deployment "__DEPLOYMENT__" always; + location / { try_files $uri $uri/ =404; } + location = /healthz { default_type text/plain; return 200 "omnisdk __DEPLOYMENT__ ready\n"; } + } +} +NGINX +cat > /usr/share/nginx/html/index.html <<'HTML' +omnisdk nginx

omnisdk __DEPLOYMENT__

nginx, provisioned through merged EC2 documents.

+HTML +nginx -t +systemctl enable --now nginx diff --git a/docs/provider_sessions.md b/docs/provider_sessions.md index a4838454..776ab9f0 100644 --- a/docs/provider_sessions.md +++ b/docs/provider_sessions.md @@ -187,3 +187,18 @@ With caller access management enabled, variable reads require `session/variables resets require `session/config:write`. Per-variable/per-secret owner and group grants are not yet implemented. Resolved SQL objects are checked before execution; variable substitution grants no additional provider/backend permission. Existing credential reload remains a separate mechanism. + +## Session document merge + +```sql +SET "$.document_merge" = '{"patches":[{"service":"stackql_unstable_google.compute","merge":{"components":{"x-stackQL-resources":{"networks":{"methods":{"list":{"response":{"objectKey":"$.items"}}}}}}}}]}'; +-- Alternatively, read the same configuration from a server-local file: +SET "$.document_merge" = '@/absolute/path/merge.json'; +UNSET "$.document_merge"; +``` + +Requires the document-driven provider/preview integration. Subsequent document descriptions, queries and IaC use the effective view; the base registry is unchanged. Each SET replaces the patch set against the base (include all desired fragments in order). Failed SET preserves the previous view; UNSET restores the base. New client sessions inherit the configured view but later changes are isolated. + +The default merge cache is `/cache/document-views` (normally under `.stackql/`); `doc_cache.dir` overrides it. IaC ledger and run journals already default to `/iac`. + +Keep the base immutable and the cache available while readers use it. `fresh` is rejected; use a new cache directory instead. File paths refer to the StackQL process's filesystem, not a remote SQL client's filesystem. JSON objects may also be supplied through session variables. Inline JSON follows SQL string escaping; use `@file` to avoid escaping embedded programs. diff --git a/go.mod b/go.mod index 77e8fd8a..62de6875 100644 --- a/go.mod +++ b/go.mod @@ -7,7 +7,7 @@ require ( github.com/Masterminds/semver v1.4.2 github.com/PaesslerAG/gval v1.0.0 github.com/PaesslerAG/jsonpath v0.1.1 - github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e + github.com/chzyer/readline v1.5.1 github.com/golang-jwt/jwt/v5 v5.3.1 github.com/google/go-jsonnet v0.17.0 github.com/google/licensecheck v0.3.1 @@ -23,7 +23,7 @@ require ( github.com/spf13/cobra v1.10.2 github.com/spf13/pflag v1.0.10 github.com/spf13/viper v1.10.1 - github.com/stackql-labs/omnisdk v0.1.10-beta01 + github.com/stackql-labs/omnisdk v0.1.11-alpha07 github.com/stackql/go-suffix-map v0.0.1-alpha02 github.com/stackql/psql-wire v0.1.2-beta02 github.com/stackql/stackql-parser v0.0.17-alpha05 @@ -131,6 +131,7 @@ require ( github.com/zeebo/xxh3 v1.0.2 // indirect go.opentelemetry.io/otel v1.44.0 // indirect go.opentelemetry.io/otel/trace v1.44.0 // indirect + go.starlark.net v0.0.0-20261021141710-4f4267fb140d // indirect go.uber.org/atomic v1.9.0 // indirect go.uber.org/multierr v1.7.0 // indirect go.uber.org/zap v1.21.0 // indirect diff --git a/go.sum b/go.sum index 1fbd578c..852e10ca 100644 --- a/go.sum +++ b/go.sum @@ -328,8 +328,8 @@ github.com/spf13/viper v1.10.1 h1:nuJZuYpG7gTj/XqiUwg8bA0cp1+M2mC3J4g5luUYBKk= github.com/spf13/viper v1.10.1/go.mod h1:IGlFPqhNAPKRxohIzWpI5QEy4kuI7tcl5WvR+8qy1rU= github.com/spiffe/go-spiffe/v2 v2.7.0 h1:uXe1MflJoHw58wAUvxVlcM7WpKtijWG7I1UidcGh6g4= github.com/spiffe/go-spiffe/v2 v2.7.0/go.mod h1:47Q0Q9/AqGha8QLHp+kxpH4Wca7X7EnOtlIJy3mxZ3U= -github.com/stackql-labs/omnisdk v0.1.10-beta01 h1:jpIiIfcpC5hm0MnBNxBz8iHNaHTFmvt0Xlcpw61RqMI= -github.com/stackql-labs/omnisdk v0.1.10-beta01/go.mod h1:pwxphb3OMfEui7uCMmReoB+ABshSBl83dszn4Sf4vcg= +github.com/stackql-labs/omnisdk v0.1.11-alpha07 h1:zRfjdx+ID0TkeccvAW0z4pysxHI8JHgt9eFsphFOcOM= +github.com/stackql-labs/omnisdk v0.1.11-alpha07/go.mod h1:G6qZy+6Vbuy9sSfws6erzpbbuThI+cD/uXDQPgQkRBI= github.com/stackql/go-suffix-map v0.0.1-alpha02 h1:JI4aiq/OfHW4a/BiNqK31NbyaR94zlUOhGZof3G/OUo= github.com/stackql/go-suffix-map v0.0.1-alpha02/go.mod h1:6BSMz4K/gIYfqdVRb7V8xL165wiPYyCjD9Kx7hLMd9Y= github.com/stackql/psql-wire v0.1.2-beta02 h1:7DfliBpyyCaUHVJ1Jxk37q3zVwVLyh6+FCHeM0/gbAQ= @@ -378,6 +378,8 @@ go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRk go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA= go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk= go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE= +go.starlark.net v0.0.0-20261021141710-4f4267fb140d h1:wmJ4xTYwnBup3nwA+3OtmpaCwd90YXVyyCHZQoiD0+0= +go.starlark.net v0.0.0-20261021141710-4f4267fb140d/go.mod h1:Iue6g6iirlfLoVi/DYCi5/x0h/bAOuWF3dULTKpt2Vo= go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= diff --git a/internal/stackql/handler/document_merge.go b/internal/stackql/handler/document_merge.go new file mode 100644 index 00000000..8de05f2a --- /dev/null +++ b/internal/stackql/handler/document_merge.go @@ -0,0 +1,83 @@ +package handler + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/stackql-labs/omnisdk/pkg/omnisdk" + "gopkg.in/yaml.v2" +) + +// DocumentRegistry selects this client's effective view without changing the base registry. +func (hc *standardHandlerContext) DocumentRegistry(base string) string { + state := hc.providerSessionState() + state.mu.Lock() + defer state.mu.Unlock() + if state.documentView != "" { + return state.documentView + } + return base +} + +func (hc *standardHandlerContext) setDocumentMerge(value interface{}) error { + state := hc.providerSessionState() + state.mu.Lock() + defer state.mu.Unlock() + if value == nil { + state.documentView = "" + return nil + } + if text, ok := value.(string); ok { + raw := []byte(text) + if filename, file := strings.CutPrefix(text, "@"); file { + data, err := os.ReadFile(filename) + if err != nil { + return fmt.Errorf("document_merge: %w", err) + } + raw = data + } + if err := json.Unmarshal(raw, &value); err != nil { + return fmt.Errorf("document_merge: invalid JSON: %w", err) + } + } + var config struct { + Patches []omnisdk.DocPatch `json:"patches"` + Cache omnisdk.DocCache `json:"doc_cache"` + } + if err := decodeSessionObject(value, &config); err != nil { + return err + } + if len(config.Patches) == 0 { + return fmt.Errorf("document_merge requires patches; use UNSET to restore the base") + } + if config.Cache.Fresh { + return fmt.Errorf("document_merge: fresh is unsupported; use a new cache directory") + } + if config.Cache.Dir == "" { + config.Cache.Dir = filepath.Join(hc.runtimeContext.ApplicationFilesRootPath, "cache", "document-views") + } + var registry omnisdk.RegistryConfig + if err := yaml.Unmarshal([]byte(hc.runtimeContext.RegistryRaw), ®istry); err != nil { + return err + } + base := registry.LocalDocRoot + if base == "" { + base = hc.runtimeContext.ApplicationFilesRootPath + if strings.HasPrefix(registry.RegistryURL, "file:") { + base = filepath.Clean(filepath.Join(strings.TrimPrefix(registry.RegistryURL, "file:"), "..")) + } + } + prefix := "src" + if registry.SrcPrefix != nil { + prefix = *registry.SrcPrefix + } + view, err := omnisdk.EffectiveRegistry(filepath.Join(base, prefix), config.Patches, config.Cache) + if err != nil { + return fmt.Errorf("document_merge: %w", err) + } + state.documentView = view + return nil +} diff --git a/internal/stackql/handler/document_merge_test.go b/internal/stackql/handler/document_merge_test.go new file mode 100644 index 00000000..ff26ba85 --- /dev/null +++ b/internal/stackql/handler/document_merge_test.go @@ -0,0 +1,74 @@ +package handler //nolint:testpackage // Exercise private session configuration boundaries. + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" +) + +func TestDocumentMergeSessionBoundary(t *testing.T) { + hc := sessionTestHandler() + app := t.TempDir() + base := filepath.Join(app, "registry", "src") + service := "widgets/v00.00.00000/services/things.yaml" + write := func(name, body string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(name), 0755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(name, []byte(body), 0600); err != nil { + t.Fatal(err) + } + } + write(filepath.Join(base, "widgets/v00.00.00000/provider.yaml"), `{"id":"widgets","name":"widgets","version":"v00.00.00000","providerServices":{"things":{"id":"things","name":"things","service":{"$ref":"widgets/v00.00.00000/services/things.yaml"}}}}`) + write(filepath.Join(base, service), `{"openapi":"3.0.0","info":{"title":"original","version":"v1"}}`) + hc.runtimeContext.ApplicationFilesRootPath = app + raw, _ := json.Marshal(map[string]string{"localDocRoot": filepath.Join(app, "registry")}) + hc.runtimeContext.RegistryRaw = string(raw) + config := `{"patches":[{"service":"stackql_unstable_widgets.things","merge":{"info":{"title":"patched"}}}]}` + for name, file := range map[string]bool{"inline": false, "file": true} { + t.Run(name, func(t *testing.T) { + value := config + if file { + name := filepath.Join(app, "merge.json") + write(name, config) + value = "@" + name + } + checkDocumentMergeBoundary(t, hc, app, base, service, value) + }) + } +} + +func checkDocumentMergeBoundary(t *testing.T, hc *standardHandlerContext, app, base, service, value string) { + t.Helper() + if err := hc.SetConfigAtPath("document_merge", value, "session"); err != nil { + t.Fatal(err) + } + view := hc.DocumentRegistry(base) + if filepath.Dir(view) != filepath.Join(app, "cache", "document-views") { + t.Fatalf("unexpected default cache: %s", view) + } + original, _ := os.ReadFile(filepath.Join(base, service)) + if string(original) != `{"openapi":"3.0.0","info":{"title":"original","version":"v1"}}` { + t.Fatal("base changed") + } + for _, invalid := range []string{`{`, `{"patches":[]}`, `{"typo":true}`, `{"patches":[{"service":"missing.things","merge":{}}]}`, `{"patches":[{"service":"stackql_unstable_widgets.things","merge":{}}],"doc_cache":{"fresh":true}}`, "@" + filepath.Join(app, "missing.json")} { + if err := hc.SetConfigAtPath("document_merge", invalid, "session"); err == nil { + t.Fatalf("accepted %s", invalid) + } + if hc.DocumentRegistry(base) != view { + t.Fatal("failed SET changed view") + } + } + other := sessionTestHandler() + if other.DocumentRegistry(base) != base { + t.Fatal("view leaked") + } + if err := hc.SetConfigAtPath("document_merge", nil, "session"); err != nil { + t.Fatal(err) + } + if hc.DocumentRegistry(base) != base { + t.Fatal("UNSET failed") + } +} diff --git a/internal/stackql/handler/handler.go b/internal/stackql/handler/handler.go index a1416da6..1137997c 100644 --- a/internal/stackql/handler/handler.go +++ b/internal/stackql/handler/handler.go @@ -562,6 +562,9 @@ func (hc *standardHandlerContext) SetConfigAtPath(path string, rhs interface{}, if strings.HasPrefix(path, "variables.") { return hc.setSessionVariable(path, rhs) } + if path == "document_merge" { + return hc.setDocumentMerge(rhs) + } if handled, err := hc.setProviderSessionConfig(path, rhs, scope); handled { return err } diff --git a/internal/stackql/handler/provider_session.go b/internal/stackql/handler/provider_session.go index 9cdbfcc2..af77773b 100644 --- a/internal/stackql/handler/provider_session.go +++ b/internal/stackql/handler/provider_session.go @@ -19,12 +19,13 @@ import ( ) type providerSessionState struct { - variables map[string]any - base omnisdk.Args - mu sync.Mutex - client omnisdk.ClientSession - store secrets.Store - bindings map[string]omnisdk.ProviderBinding + variables map[string]any + base omnisdk.Args + mu sync.Mutex + client omnisdk.ClientSession + store secrets.Store + bindings map[string]omnisdk.ProviderBinding + documentView string } func decodeSessionObject(value interface{}, target interface{}) error { @@ -304,6 +305,7 @@ func (source *standardHandlerContext) NewClientContext() HandlerContext { source.runtimeContext.APIRequestTimeout, source.runtimeContext.HTTPPageLimit, ) target.providerSession.variables = maps.Clone(state.variables) + target.providerSession.documentView = state.documentView target.providerSession.bindings = make(map[string]omnisdk.ProviderBinding, len(state.bindings)) for alias, binding := range state.bindings { binding.Params = maps.Clone(binding.Params) diff --git a/internal/stackql/intrinsic/doc.go b/internal/stackql/intrinsic/doc.go index a9685312..7877826f 100644 --- a/internal/stackql/intrinsic/doc.go +++ b/internal/stackql/intrinsic/doc.go @@ -82,16 +82,7 @@ func localDocRoot(runtimeCtx dto.RuntimeCtx) string { // name carries a dot ("googleapis.com") cannot be named that way. func docRoot(ctx queryContext, bundle string) (string, error) { bundle = strings.TrimPrefix(providerAddress(ctx, bundle), UnstablePrefix) - runtime := ctx.GetRuntimeContext() - prefix := "src" - var registryCfg omnisdk.RegistryConfig - if err := yaml.Unmarshal([]byte(runtime.RegistryRaw), ®istryCfg); err != nil { - return "", err - } - if registryCfg.SrcPrefix != nil { - prefix = *registryCfg.SrcPrefix - } - root := filepath.Join(localDocRoot(runtime), prefix, bundleDir(bundle)) + root := filepath.Join(registryRoot(ctx), bundleDir(bundle)) matches, err := filepath.Glob(filepath.Join(root, "*", "provider.yaml")) if err != nil || len(matches) == 0 { return "", fmt.Errorf("no provider document for '%s%s' under '%s'", UnstablePrefix, bundle, root) diff --git a/internal/stackql/intrinsic/iac.go b/internal/stackql/intrinsic/iac.go index 8ce8418a..09d5fafb 100644 --- a/internal/stackql/intrinsic/iac.go +++ b/internal/stackql/intrinsic/iac.go @@ -51,6 +51,10 @@ func iacSelectFunc( switch { case strings.EqualFold(resource, convergeRelation): action = convergeSelectFunc(ctx, node) + case strings.EqualFold(resource, recoverRelation): + return recoverSelectFunc(ctx, node), true + case strings.EqualFold(resource, ledgerRelation): + return ledgerSelectFunc(ctx, node), true case strings.EqualFold(resource, destroyRelation): action = destroySelectFunc(ctx, node) default: @@ -382,6 +386,17 @@ func blueprintSelectFunc(resource string) func() internaldto.ExecutorOutput { // iacBindings keeps typed values separate from provider execution scope. func iacBindings(params map[string]string) (map[string]any, map[string]any, error) { + file := popPredicate(params, "inputs_file") + if file != "" { + if _, inline := params["inputs"]; inline { + return nil, nil, fmt.Errorf("supply inputs or inputs_file, not both") + } + raw, err := os.ReadFile(file) + if err != nil { + return nil, nil, fmt.Errorf("inputs_file: %w", err) + } + params["inputs"] = string(raw) + } var maps [2]map[string]any for index, name := range []string{"inputs", "environment"} { raw := popPredicate(params, name) @@ -426,7 +441,11 @@ func (r *iacRows) Next() bool { } if failed { r.mu.Lock() - r.failure = iacOutcomeError{key: fmt.Sprint(row["key"]), status: status} + failure := iacOutcomeError{key: fmt.Sprint(row["key"]), status: status} + if detail := row["error"]; detail != nil && fmt.Sprint(detail) != "" { + failure.cause = fmt.Errorf("IaC operation failed for key %s: %s", failure.key, detail) + } + r.failure = failure r.mu.Unlock() } return true diff --git a/internal/stackql/intrinsic/iac_ledger.go b/internal/stackql/intrinsic/iac_ledger.go new file mode 100644 index 00000000..285f66a5 --- /dev/null +++ b/internal/stackql/intrinsic/iac_ledger.go @@ -0,0 +1,55 @@ +package intrinsic + +import ( + "context" + "fmt" + + "github.com/stackql-labs/omnisdk/pkg/omnisdk" + "github.com/stackql/stackql-parser/go/vt/sqlparser" + "github.com/stackql/stackql/internal/stackql/internal_data_transfer/internaldto" +) + +// A capability check permits a clear upgrade error with an older published SDK. +type ledgerReader interface { + Read(context.Context) ([]omnisdk.Row, error) +} + +type ledgerReadPlan struct { + reader ledgerReader + resource string +} + +func (p ledgerReadPlan) Open(ctx context.Context) (omnisdk.Rows, error) { + rows, err := p.reader.Read(ctx) + if err != nil { + return nil, err + } + if p.resource != "" { + selected := make([]omnisdk.Row, 0, 1) + for _, row := range rows { + if row["resource_key"] == p.resource { + selected = append(selected, row) + } + } + rows = selected + } + return &materializedRelationRows{ctx: ctx, rows: rows}, nil +} + +func ledgerSelectFunc(ctx queryContext, node *sqlparser.Select) func() internaldto.ExecutorOutput { + params, config, refusal := ledgerPredicates(ctx, node, relationAddress(iacService, ledgerRelation)) + if refusal != nil { + return refusal + } + resource := popPredicate(params, "resource_key") + if len(params) != 0 { + return refuse(fmt.Errorf("ledger accepts collection, state and optional resource_key predicates")) + } + reader, ok := config.(ledgerReader) + if !ok { + return refuse(fmt.Errorf("the Omni dependency needs the read-only ledger capability; upgrade Omni")) + } + return func() internaldto.ExecutorOutput { + return streamPlan(ctx, ledgerReadPlan{reader: reader, resource: resource}, ledgerRelation, node.SelectExprs) + } +} diff --git a/internal/stackql/intrinsic/iac_recover.go b/internal/stackql/intrinsic/iac_recover.go new file mode 100644 index 00000000..3bf220d7 --- /dev/null +++ b/internal/stackql/intrinsic/iac_recover.go @@ -0,0 +1,63 @@ +package intrinsic + +import ( + "context" + "fmt" + + "github.com/stackql-labs/omnisdk/pkg/omnisdk" + "github.com/stackql/stackql-parser/go/vt/sqlparser" + "github.com/stackql/stackql/internal/stackql/internal_data_transfer/internaldto" +) + +type ledgerRecoverer interface { + Recover(context.Context, string, string, string, string, omnisdk.Args) ([]omnisdk.Row, error) +} + +type ledgerRecoveryPlan struct { + recoverer ledgerRecoverer + resource, outcome, identity, version string + args omnisdk.Args +} + +func (p ledgerRecoveryPlan) Open(ctx context.Context) (omnisdk.Rows, error) { + rows, err := p.recoverer.Recover(ctx, p.resource, p.outcome, p.identity, p.version, p.args) + if err != nil { + return nil, err + } + return &materializedRelationRows{ctx: ctx, rows: rows}, nil +} + +func recoverSelectFunc(ctx queryContext, node *sqlparser.Select) func() internaldto.ExecutorOutput { + params, config, refusal := ledgerPredicates(ctx, node, relationAddress(iacService, recoverRelation)) + if refusal != nil { + return refusal + } + resource := popPredicate(params, "resource_key") + outcome := popPredicate(params, "outcome") + identity := popPredicate(params, "identity") + version := popPredicate(params, "version") + if len(params) != 0 || resource == "" || version == "" || (outcome != "absent" && outcome != "applied") { + return refuse(fmt.Errorf("recovery requires resource_key, version and verified outcome absent or applied; " + + "applied also requires identity")) + } + recoverer, ok := config.(ledgerRecoverer) + if !ok { + return refuse(fmt.Errorf("upgrade Omni to the release providing explicit ledger recovery")) + } + return func() internaldto.ExecutorOutput { + args := omnisdk.Args{} + err := configureCaller(ctx, &args) + if err != nil { + return internaldto.NewErroneousExecutorOutput(err) + } + plan := ledgerRecoveryPlan{ + recoverer: recoverer, resource: resource, outcome: outcome, + identity: identity, version: version, args: args, + } + output := streamPlan(ctx, plan, recoverRelation, node.SelectExprs) + if outputErr := output.GetError(); outputErr != nil { + return internaldto.NewErroneousExecutorOutput(iacOutcomeError{cause: outputErr}) + } + return output + } +} diff --git a/internal/stackql/intrinsic/iac_test.go b/internal/stackql/intrinsic/iac_test.go index e77f1f96..bcba7d95 100644 --- a/internal/stackql/intrinsic/iac_test.go +++ b/internal/stackql/intrinsic/iac_test.go @@ -5,6 +5,7 @@ import ( "errors" "os" "path/filepath" + "strings" "testing" "github.com/stackql-labs/omnisdk/pkg/omnisdk" @@ -156,6 +157,9 @@ func TestIaCReportedOutcomesAreApplicationFailures(t *testing.T) { if cursor.Err() == nil { t.Fatalf("unsuccessful outcome accepted: %v", row) } + if detail, ok := row["error"].(string); ok && !strings.Contains(cursor.Err().Error(), detail) { + t.Fatalf("underlying failure discarded: %v", cursor.Err()) + } } cursor := &iacRows{Rows: &fakeRows{rows: []omnisdk.Row{{"status": "applied"}}}} for cursor.Next() { @@ -164,3 +168,24 @@ func TestIaCReportedOutcomesAreApplicationFailures(t *testing.T) { t.Fatal(err) } } + +func TestIaCInputsFile(t *testing.T) { + file := filepath.Join(t.TempDir(), "inputs.json") + if err := os.WriteFile(file, []byte(`{"large":9007199254740993,"flag":true}`), 0600); err != nil { + t.Fatal(err) + } + params := map[string]string{"inputs_file": file, "region": "east"} + inputs, _, err := iacBindings(params) + if err != nil { + t.Fatal(err) + } + if inputs["large"] != json.Number("9007199254740993") || inputs["flag"] != true || len(params) != 1 { + t.Fatal(inputs, params) + } + if _, _, err := iacBindings(map[string]string{"inputs_file": file, "inputs": "{}"}); err == nil { + t.Fatal("accepted conflicting inputs") + } + if _, _, err := iacBindings(map[string]string{"inputs_file": file + ".missing"}); err == nil { + t.Fatal("accepted missing file") + } +} diff --git a/internal/stackql/intrinsic/operation_completion.go b/internal/stackql/intrinsic/operation_completion.go index 2bfa06c8..d28bbe56 100644 --- a/internal/stackql/intrinsic/operation_completion.go +++ b/internal/stackql/intrinsic/operation_completion.go @@ -245,7 +245,16 @@ func completionAddress(catalogue aot.Catalog, service, link string) (string, map return "", nil, fmt.Errorf("AWAIT link does not match a declared GET operation in %s", service) } +// acceptedMutation preserves plain SQL's return-on-acceptance contract. +type acceptedMutation struct{} + +func (acceptedMutation) Complete(_ context.Context, _, _ string, _ map[string]any, row omnisdk.Row) (omnisdk.Row, error) { + return row, nil +} + func configureMutationCompletion(ctx queryContext, translated docQuery, args *omnisdk.Args) error { + // SQL owns AWAIT semantics; managed IaC retains document lifecycle waits. + args.MutationCompletion = acceptedMutation{} if !queryAwaitsMutation(translated) { return nil } diff --git a/internal/stackql/intrinsic/preview_services.go b/internal/stackql/intrinsic/preview_services.go index 13ec7a0e..d31a53ee 100644 --- a/internal/stackql/intrinsic/preview_services.go +++ b/internal/stackql/intrinsic/preview_services.go @@ -37,6 +37,8 @@ const ( graphRelation = "query" convergeRelation = "run" destroyRelation = "destroy" + ledgerRelation = "ledger" + recoverRelation = "recover" ) const ( @@ -110,7 +112,11 @@ func registryRoot(ctx queryContext) string { if err := yaml.Unmarshal([]byte(runtime.RegistryRaw), &cfg); err == nil && cfg.SrcPrefix != nil { prefix = *cfg.SrcPrefix } - return filepath.Join(localDocRoot(runtime), prefix) + root := filepath.Join(localDocRoot(runtime), prefix) + if host, ok := ctx.(interface{ DocumentRegistry(string) string }); ok { + return host.DocumentRegistry(root) + } + return root } // popPredicate takes a predicate out of the parameter map. What remains after @@ -324,6 +330,14 @@ func previewTables(service string) []table { isData: true, description: "every resource a named collection holds, deleted", }} + out = append(out, table{ + service: iacService, name: recoverRelation, isData: true, + description: "explicit recovery of an operator-verified pending outcome", + }) + out = append(out, table{ + service: iacService, name: ledgerRelation, isData: true, + description: "read-only resource identities and phases from the collection ledger", + }) return append(out, blueprintTables()...) } @@ -344,7 +358,8 @@ func describePreviewTableFunc( resource := node.Table.Name.GetRawVal() extended := isExtended(node.Extended) if service == iacService && !strings.EqualFold(resource, convergeRelation) && - !strings.EqualFold(resource, destroyRelation) { + !strings.EqualFold(resource, destroyRelation) && !strings.EqualFold(resource, ledgerRelation) && + !strings.EqualFold(resource, recoverRelation) { blueprint, ok := blueprintFor(resource) if !ok { return refuse(fmt.Errorf( diff --git a/internal/stackql/planbuilder/plan_builder.go b/internal/stackql/planbuilder/plan_builder.go index 3111f4e9..84788240 100644 --- a/internal/stackql/planbuilder/plan_builder.go +++ b/internal/stackql/planbuilder/plan_builder.go @@ -232,7 +232,7 @@ func setLogic(pbi planbuilderinput.PlanBuilderInput, setExpr *sqlparser.SetExpr) lhsTrimmed = strings.TrimPrefix(strings.TrimPrefix(lhsTrimmed, "$"), ".") root, _, _ := strings.Cut(lhsTrimmed, ".") switch root { - case "auth", "providers", "secret_store", "secrets": + case "auth", "providers", "secret_store", "secrets", "document_merge": default: return fmt.Errorf("unknown intrinsic session configuration %q", lhsTrimmed) } diff --git a/test/python/stackql_test_tooling/access_management_keywords.py b/test/python/stackql_test_tooling/access_management_keywords.py index 1655c25d..3adcf72f 100644 --- a/test/python/stackql_test_tooling/access_management_keywords.py +++ b/test/python/stackql_test_tooling/access_management_keywords.py @@ -391,6 +391,9 @@ def denied(sql, credential=None, concurrent=False): if scenario == "policy": for sql in ["DELETE FROM stackql_unstable_accessdemo.service.items;", "INSERT INTO stackql_unstable_accessdemo.service.items (id) VALUES ('new');", "EXEC stackql_unstable_accessdemo.service.items.create;", "SELECT * FROM stackql_preview.iac.run WHERE collection = 'test' AND resources = '[]';", "SELECT * FROM stackql_preview.iac.destroy WHERE collection = 'test';", "SELECT * FROM stackql_access_principals;", "SELECT pg_read_file('/etc/passwd');"]: denied(sql, reader) + # Every IaC relation inherits the AST gate, including new/unclassified commands. + for relation in ["ledger", "recover", "unlock", "not_a_published_operation"]: + denied("SELECT * FROM stackql_preview.iac." + relation + " WHERE collection = 'test';", reader) denied(query, token(subject="denied")) with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: futures = [executor.submit(success, query, reader, "known-item"), executor.submit(denied, "SELECT 1", token(subject="denied"), True)] @@ -422,6 +425,11 @@ def converge(label, keys): assert all(item.get("name") == label for item in managed.values()), "IaC reported applied without converging desired values" converge("first", ["one", "two"]) assert len(managed) == 2, "IaC create did not retain both resources" + before = counts.copy() + ledger_query = "SELECT identity, phase FROM stackql_preview.iac.ledger WHERE collection = 'acceptance' AND state = 'audit' AND resource_key = 'one';" + success(ledger_query, operator, "live") + denied(ledger_query, reader) + assert counts == before, "ledger authorization triggered provider effects" db.execute("INSERT INTO stackql_access_principals VALUES ('operator2','{}')") db.execute("INSERT INTO stackql_access_identities VALUES ('access-test','operator2','operator2')") db.execute("INSERT INTO stackql_access_memberships VALUES ('operator2','operators')") diff --git a/test/robot/functional/ProviderSessionLibrary.py b/test/robot/functional/ProviderSessionLibrary.py index 0a478303..225511ad 100644 --- a/test/robot/functional/ProviderSessionLibrary.py +++ b/test/robot/functional/ProviderSessionLibrary.py @@ -623,3 +623,58 @@ def string_boolean_integer_and_null_variables_filter_provider_rows(self): sql += "SET enabled = false; SELECT id FROM typed.things.widgets WHERE region = '42' AND active = $enabled;" result = self.run_sql(sql, [('Bearer typed-token', '42'), ('Bearer typed-token', '42')]) assert result.stdout.rstrip().endswith('null'), result.stdout + + def document_merge_inline_file_and_reset(self): + binding = self.setting("providers.merged", {"provider": "widgets", "tenancy": "merge-test", "auth": {"type": "bearer", "credentials": "merge-token"}, "params": {"region": "merge-region"}}) + config = {"patches": [{"service": "stackql_unstable_widgets.things", "merge": {"components": {"x-stackQL-resources": {"widgets": {"methods": {"list": {"response": {"transform": {"type": "golang_template_json_v0.3.0", "body": '[{"id":"patched"}]'}}}}}}}}}]} + query = "SELECT id FROM merged.things.widgets;" + patch_file = self.root / "merge.json" + patch_file.write_text(json.dumps(config)) + base_doc = self.root / "registry/src/widgets/v00.00.00000/services/things.yaml" + original = base_doc.read_bytes() + for value in [json.dumps(config).replace("\\", "\\\\"), "@" + str(patch_file)]: + sql = binding + self.setting("document_merge", value) + query + sql += 'UNSET "$.document_merge";' + query + result = self.run_sql(sql, [("Bearer merge-token", "merge-region")] * 2) + assert 'patched' in result.stdout and 'merge-region' in result.stdout, result.stdout + assert base_doc.read_bytes() == original, "base document changed" + result = self.run_sql(binding + query, [("Bearer merge-token", "merge-region")]) + assert 'patched' not in result.stdout, "merge leaked into a new session" + + def iac_inputs_file_errors_precede_provider_dispatch(self): + missing = str(self.root / "missing-inputs.json") + sql = "SELECT * FROM stackql_preview.iac.run WHERE collection = 'file-check' AND resources = '[]' AND inputs_file = '" + missing + "';" + before = len(self.calls) + result = subprocess.run(self.prefix + ["exec", sql] + self.flags, text=True, capture_output=True, timeout=30, env=self.env) + assert result.returncode != 0 and "inputs_file:" in result.stderr, result.stderr + assert len(self.calls) == before, "provider called before inputs loaded" + + def ledger_query_is_read_only_and_collection_scoped(self): + import base64 + def encoded(value): + return base64.b64encode(value.encode()).decode() + root = self.root / "app/iac/ledger" + for collection in ["demo", "demo-other"]: + directory = root / collection + directory.mkdir(parents=True, exist_ok=True) + (directory / "vm.v1.json").write_text(json.dumps({"phase": 1, "identity": encoded("i-ledger-test"), "about": encoded(json.dumps({"provider":"aws","address":"ec2.web_instance","params":{"secret":"hidden"}}))})) + before = {str(p.relative_to(root)): p.read_bytes() for p in root.rglob("*") if p.is_file()} + result = self.run_sql("SELECT * FROM stackql_preview.iac.ledger WHERE collection = 'demo' AND resource_key = 'vm';", []) + assert "i-ledger-test" in result.stdout and 'live' in result.stdout, result.stdout + assert "demo-other" not in result.stdout and "hidden" not in result.stdout, result.stdout + after = {str(p.relative_to(root)): p.read_bytes() for p in root.rglob("*") if p.is_file()} + assert before == after, "ledger changed during inspection" + assert not (self.root / "app/iac/journal").exists(), "reader created a journal" + + def recover_verified_absent_create_without_provider_calls(self): + import base64 + root = self.root / "app/iac/ledger/recover-test" + root.mkdir(parents=True, exist_ok=True) + record = {"phase":0,"proposed":base64.b64encode(b"{}").decode()} + (root / "pending.v1.json").write_text(json.dumps(record)) + sql = "SELECT * FROM stackql_preview.iac.recover WHERE collection = 'recover-test' AND resource_key = 'pending' AND outcome = 'absent' AND version = '1';" + result = self.run_sql(sql, []) + assert 'recovered_absent' in result.stdout, result.stdout + assert json.loads((root / 'pending.v2.json').read_text())['tombstone'], "recovery did not clear verified absent create" + evidence = [json.loads(line) for path in (self.root / 'app/iac/journal').glob('*.diagnostics.jsonl') for line in path.read_text().splitlines()] + assert any(row.get('diagnostic', {}).get('event') == 'recovery_assertion' for row in evidence), 'recovery decision not durable' diff --git a/test/robot/functional/stackql_mocked_from_cmd_line.robot b/test/robot/functional/stackql_mocked_from_cmd_line.robot index 11d609a5..deb89011 100644 --- a/test/robot/functional/stackql_mocked_from_cmd_line.robot +++ b/test/robot/functional/stackql_mocked_from_cmd_line.robot @@ -8979,8 +8979,21 @@ Insert Returning Simple Projection ... stdout=${CURDIR}/tmp/Insert-Returning-Simple-Projection.tmp ... stderr=${CURDIR}/tmp/Insert-Returning-Simple-Projection-stderr.tmp +Google Plain Insert Does Not Inherit IaC Self Read Wait + [Documentation] A plain SQL insert returns on acceptance despite Google's embedded lifecycle default. + Should StackQL Exec Inline Equal Stderr + ... ${STACKQL_EXE} + ... ${OKTA_SECRET_STR} + ... ${GITHUB_SECRET_STR} + ... ${K8S_SECRET_STR} + ... ${REGISTRY_NO_VERIFY_CFG_STR} + ... ${AUTH_CFG_STR} + ... ${SQL_BACKEND_CFG_STR_CANONICAL} + ... insert into google.compute.networks(project, data__name, data__autoCreateSubnetworks) select 'mutable-project', 'auto-test-01', false; + ... The operation was despatched successfully + Insert Async Returning Simple Projection - [Documentation] Insert a row into a table and return projected new object values. For **asynchronously** created objects. + [Documentation] AWAIT polls the operation before reading the resource; embedded IaC defaults must not add a self-read first. ${inputStr} = Catenate ... insert /*+ AWAIT */ into google.compute.networks(project, data__name, data__autoCreateSubnetworks) select 'mutable-project', 'auto-test-01', false returning creationTimestamp, name; ${outputStr} = Catenate SEPARATOR=\n @@ -11875,6 +11888,8 @@ Preview Iac Show Resources Lists Run And Blueprint Handles ... name,id ... run,stackql_preview.iac.run ... destroy,stackql_preview.iac.destroy + ... recover,stackql_preview.iac.recover + ... ledger,stackql_preview.iac.ledger ... aws_vpc_subnet,stackql_preview.iac.aws_vpc_subnet Should StackQL Exec Inline Equal ... ${STACKQL_EXE} diff --git a/test/robot/functional/stackql_provider_sessions.robot b/test/robot/functional/stackql_provider_sessions.robot index 1bd5cd6f..a1e36ee6 100644 --- a/test/robot/functional/stackql_provider_sessions.robot +++ b/test/robot/functional/stackql_provider_sessions.robot @@ -83,3 +83,15 @@ Variable Identifiers Work In CTEs Joins And Subqueries String Boolean Integer And Null Variables Filter Provider Rows String Boolean Integer And Null Variables Filter Provider Rows + +Document Merge Supports Inline JSON File Input And Reset + Document Merge Inline File And Reset + +IaC Inputs File Errors Precede Provider Dispatch + Iac Inputs File Errors Precede Provider Dispatch + +Ledger Query Is Read Only And Collection Scoped + Ledger Query Is Read Only And Collection Scoped + +Recover Verified Absent Create Without Provider Calls + Recover Verified Absent Create Without Provider Calls