[LTP] [PATCH 1/1] ci: Add hook to mirror docparse to homepage

[Petr Vorel]

> Hi Peter, two comments:

> +  metadata-mirror:
> +    permissions:
> +      contents: write # for git push

> Since you are using GH_PERSONAL_ACCESS_TOKEN and not the default
> GITHUB_TOKEN for push, I think "contents: read" should be sufficient. Could
> you please try it?

Sure, I'll test it.

> +      - name: Checkout LTP homepage
> +        uses: actions/checkout@v3
> +        with:
> +          repository: "linux-test-project/linux-test-project.github.com"
> +          path: linux-test-project.github.com
> +          persist-credentials: false
> +        env:
> +          GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}

> I think the GH_PERSONAL_ACCESS_TOKEN env variable here is not used and can
> be deleted. The checkout uses the standard GITHUB_TOKEN with read access to
> public repos.

You're most likely right, testing!

Kind regards,
Petr

> Otherwise LGTM

> On Fri, Dec 9, 2022 at 11:04 AM Petr Vorel <pvorel@suse.cz> wrote:

> > GitHub Actions git push hook generates metadata HTML and push it wo LTP
> > homepage.

> > Hook pushes only if there are actual changes in generated doc.

> > NOTE: This requires to add Personal access tokens (classic)
> > named as GH_PERSONAL_ACCESS_TOKEN with permissions to public_repo.

> > Signed-off-by: Petr Vorel <pvorel@suse.cz>
> > ---
> > Hi,

> > automatic updates of our docparse documentation to our homepage [1]
> > (linked in "nightly test documentation" [2]).

> > This does push to our homepage git repository [3].

> > This patchset required to add:

> > 1) Personal Access Token (PAT) to any developer which has write access
> > to homepage git repository [3]. In Developer settings -> Personal access
> > tokens -> Tokens (classic) [4]), where set:
> > Note: GH_PERSONAL_ACCESS_TOKEN
> > Select scopes: public_repo (minimal permission)
> > Expiration: either never or regularly renew.

> > 2) Allow PAT in LTP organisation (I dared to already set it)
> > Iin linux-test-project group -> Settings -> Third-party Access -> Personal
> > access tokens -> Settings [5]
> > select:
> > Allow access via personal access tokens (classic)
> > API and Git access will be allowed using an organization member's personal
> > access token (classic)

> > 3) Add repository action secret to ltp repository
> > IN Settings -> Actions -> New repository secret [6]:
> > name: GH_PERSONAL_ACCESS_TOKEN
> > value: the value of previously created token.

> > I haven't tested fine-grained personal access tokens, maybe they are
> > safer. Using proper GitHub App would be safer (would not include PAT),
> > but I have spent already quite a lot of time with this, thus unless
> > anybody objects I'd merge this way.

> > I did example of this run on organisation which I created just for
> > testing purposes:

> > * working example
> > action run [7], resulted in commit [8]. Example of unrelated commit [9]
> > which does not trigger doc generation [10].

> > * example of error, when token not set
> > https://github.com/foo-pevik/ltp_foo/actions/runs/3650567244
> > GH_PERSONAL_ACCESS_TOKEN environment variable is not set

> > * example of invalid GH_PERSONAL_ACCESS_TOKEN

> > https://github.com/foo-pevik/ltp_foo/actions/runs/3650579488/jobs/6166734733
> > Notice: GH_PERSONAL_ACCESS_TOKEN: ***
> > fatal: could not read Password for 'https://***@github.com': No such
> > device or address
> > Error: Process completed with exit code 128.

> > Credits:
> > * Inspiration from SwiftDocOrg [11], pointed out by Cyril
> > * Thanks a lot to Aleks for hint with "persist-credentials: false".

> > Kind regards,
> > Petr

> > [1] http://linux-test-project.github.io/
> > [2] http://linux-test-project.github.io/metadata/metadata.nightly.html
> > [3] https://github.com/linux-test-project/linux-test-project.github.com
> > [4] https://github.com/settings/tokens/new
> > [5]
> > https://github.com/organizations/linux-test-project/settings/personal-access-tokens
> > [6] https://github.com/foo-pevik/ltp_foo/settings/secrets/actions/new
> > [7] https://github.com/foo-pevik/ltp_foo/actions/runs/3651475124
> > [8]
> > https://github.com/foo-pevik/linux-test-project.github.com_foo/commit/c2b94862da2c5741f08282988dcc0302e0847a69
> > [9]
> > https://github.com/foo-pevik/ltp_foo/commit/36a66a2f304b449e1c17d72bad1b1db46a30f36f
> > [10] https://github.com/foo-pevik/ltp_foo/actions/runs/3651475552
> > [11] https://github.com/SwiftDocOrg/github-wiki-publish-action

> >  .github/workflows/metadata-mirror.yml | 141 ++++++++++++++++++++++++++
> >  1 file changed, 141 insertions(+)
> >  create mode 100644 .github/workflows/metadata-mirror.yml

> > diff --git a/.github/workflows/metadata-mirror.yml
> > b/.github/workflows/metadata-mirror.yml
> > new file mode 100644
> > index 000000000..9182c554d
> > --- /dev/null
> > +++ b/.github/workflows/metadata-mirror.yml
> > @@ -0,0 +1,141 @@
> > +# SPDX-License-Identifier: GPL-2.0-or-later
> > +# Copyright (c) 2022 Petr Vorel <pvorel@suse.cz>
> > +
> > +name: "Mirror metadata doc to homepage"
> > +
> > +on:
> > +  push:
> > +    branches:
> > +      - master
> > +
> > +permissions: {}
> > +jobs:
> > +  metadata-mirror:
> > +    permissions:
> > +      contents: write # for git push
> > +
> > +    runs-on: ubuntu-latest
> > +    if: ${{ github.repository == 'linux-test-project/ltp' }}
> > +    steps:
> > +      - name: Check secret
> > +        env:
> > +          GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN
> > }}
> > +        run: |
> > +          if [ -z "$GH_PERSONAL_ACCESS_TOKEN" ]; then
> > +            echo "::error::GH_PERSONAL_ACCESS_TOKEN environment variable
> > is not set"
> > +            exit 1
> > +          fi
> > +
> > +      - name: Checkout LTP
> > +        uses: actions/checkout@v3
> > +        with:
> > +          path: ltp
> > +          # we need to fetch whole history to get 'git describe' working
> > for correct version in docs
> > +          fetch-depth: 0
> > +
> > +      - name: Checkout LTP homepage
> > +        uses: actions/checkout@v3
> > +        with:
> > +          repository: "linux-test-project/linux-test-project.github.com"
> > +          path: linux-test-project.github.com
> > +          persist-credentials: false
> > +        env:
> > +          GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN
> > }}
> > +
> > +      - name: Check metadata need to be updated
> > +        run: |
> > +          cd "$GITHUB_WORKSPACE/ltp/testcases/"
> > +
> > +          # check for changes
> > +          cd "$GITHUB_WORKSPACE/linux-test-project.github.com"
> > +          git grep '<p><strong>Version</strong>:'
> > metadata/metadata.nightly.html
> > +          sed -ne 's/.*<p><strong>Version<\/strong>: \(.*\)<\/p>/\1/p'
> > metadata/metadata.nightly.html
> > +          old_commit=$(sed -ne 's/.*<p><strong>Version<\/strong>:
> > \(.*\)<\/p>/\1/p' metadata/metadata.nightly.html)
> > +          old_commit=$(echo "$old_commit" | sed 's/.*-g\(.*\)/\1/')
> > +          echo "::notice::old commit in linux-test-project.github.com:
> > '$old_commit'"
> > +
> > +          echo "git log $old_commit..HEAD testcases/ | wc -l"
> > +          git log $old_commit..HEAD testcases/ | wc -l
> > +
> > +          echo "git log $old_commit..HEAD testcases/ 2>&1 | wc -l"
> > +          git log $old_commit..HEAD testcases/ 2>&1 | wc -l
> > +
> > +          if [ "$(git log $old_commit..HEAD testcases/ 2>&1 | wc -l)" = 0
> > ]; then
> > +            echo "::notice::no file changes in testcases/ directory since
> > '$old_commit' => exit without commit and push"
> > +            exit 0
> > +          fi
> > +
> > +      - name: Install dependencies
> > +        run: |
> > +          apt="apt install -y --no-install-recommends"
> > +          sudo $apt asciidoctor autoconf automake libjson-perl
> > libwww-perl make
> > +
> > +          echo "which asciidoctor"
> > +          which asciidoctor || { echo "::error::missing asciidoctor";
> > exit 1; }
> > +
> > +      - name: Configure LTP
> > +        run: |
> > +          cd "$GITHUB_WORKSPACE/ltp/"
> > +          make autotools && ./configure
> > --with-metadata-generator=asciidoctor && make Version || { echo
> > "::error::LTP configure failed"; exit 1; }
> > +
> > +      - name: Generate html metadata doc
> > +        run: |
> > +          cd "$GITHUB_WORKSPACE/ltp/metadata/"
> > +          echo "going to generate metadata"
> > +          make -j$(nproc)
> > +
> > +      - name: Check possible LTP homepage update
> > +        run: |
> > +          cd "$GITHUB_WORKSPACE/ltp/"
> > +          ltp_commit=$(git log --pretty=format:"%h" -1 .)
> > +          echo "::notice::Search for $ltp_commit in
> > metadata/metadata.nightly.html"
> > +
> > +          cd "$GITHUB_WORKSPACE/linux-test-project.github.com"
> > +          grep "<p><strong>Version</strong>:"
> > metadata/metadata.nightly.html
> > +          git pull
> > +          grep "<p><strong>Version</strong>:"
> > metadata/metadata.nightly.html
> > +          if grep "<p><strong>Version</strong>: .*${ltp_commit}.*</p>"
> > metadata/metadata.nightly.html; then
> > +            echo "::notice::$ltp_commit already in
> > metadata/metadata.nightly.html => exit without commit and push"
> > +            exit 0
> > +          fi
> > +
> > +      - name: Push generated html metadata to LTP homepage
> > +        env:
> > +          GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN
> > }}
> > +        run: |
> > +          git config --global user.email "actions@github.com"
> > +          git config --global user.name "Metadata doc mirror"
> > +
> > +          cd "$GITHUB_WORKSPACE/ltp/"
> > +          commit=$(git log --pretty=format:"%h" -1 .)
> > +          commit_desc=$(git log --pretty=format:"%h (\"%s\")" -1 .)
> > +          echo "commit_desc: '$commit_desc'"
> > +
> > +          cd "$GITHUB_WORKSPACE/linux-test-project.github.com"
> > +
> > +          # pull in case previous hook pushed later than checkout here
> > had started
> > +          git pull
> > +
> > +          cp -v "$GITHUB_WORKSPACE/ltp/docparse/metadata.html"
> > metadata/metadata.nightly.html
> > +
> > +          # skip if there is only different version and time
> > +          echo "Check for changes"
> > +          if ! git diff --unified=0 --no-color | grep -Po
> > '(?<=^\+)(?!\+\+).*' | grep -v -e '^<p><strong>Version</strong>:' -e '^Last
> > updated'; then
> > +            echo "::notice::No changes in metadata/metadata.nightly.html
> > => exit without commit and push"
> > +            exit 0
> > +          fi
> > +
> > +          git add .
> > +
> > +          # only commit if there are changes
> > +          if git diff-index --quiet HEAD --; then
> > +            echo "::notice::No files changed in $GITHUB_WORKSPACE/
> > linux-test-project.github.com => exit without commit and push"
> > +            git status
> > +            exit 0
> > +          fi
> > +
> > +          printf "metadata.nightly.html: Update to $commit\n\nUpdate
> > metadata.nightly.html to $commit_desc\n" > /tmp/msg
> > +          git commit -F /tmp/msg .
> > +
> > +          echo "::notice::GH_PERSONAL_ACCESS_TOKEN:
> > $GH_PERSONAL_ACCESS_TOKEN"
> > +          git push https://${
> > GH_PERSONAL_ACCESS_TOKEN}@github.com/linux-test-project/linux-test-project.github.com.git
> > <http://GH_PERSONAL_ACCESS_TOKEN%7D@github.com/linux-test-project/linux-test-project.github.com.git>
> > --
> > 2.38.1