8391075: 'jfr assemble' doesn't handle unfinished files

Reviewed-by: mgronlun

.github/workflows/main.yml

@@ -60,7 +60,7 @@ jobs:
     runs-on: ubuntu-24.04
     env:
       # List of platforms to exclude by default
-      EXCLUDED_PLATFORMS: 'alpine-linux-x64'
+      EXCLUDED_PLATFORMS: 'alpine-linux-x64,macos-x64'
     outputs:
       linux-x64: ${{ steps.include.outputs.linux-x64 }}
       linux-x64-variants: ${{ steps.include.outputs.linux-x64-variants }}

doc/building.html

@@ -146,6 +146,7 @@ Performance</a>
 <li><a href="#virus-checking" id="toc-virus-checking">Virus
 Checking</a></li>
 <li><a href="#ccache" id="toc-ccache">Ccache</a></li>
+<li><a href="#sccache" id="toc-sccache">Sccache</a></li>
 <li><a href="#precompiled-headers"
 id="toc-precompiled-headers">Precompiled Headers</a></li>
 <li><a href="#icecc-icecream" id="toc-icecc-icecream">Icecc /
@@ -365,11 +366,13 @@ free disk space is required.</p>
 also possible to use <a href="#cross-compiling">cross-compiling</a>.</p>
 <h4 id="branch-protection">Branch Protection</h4>
 <p>In order to use Branch Protection features in the VM,
-<code>--enable-branch-protection</code> must be used. This option
-requires C++ compiler support (GCC 9.1.0+ or Clang 10+). The resulting
-build can be run on both machines with and without support for branch
-protection in hardware. Branch Protection is only supported for Linux
-targets.</p>
+<code>--enable-branch-protection</code> must be used. This option requires C++
+compiler support for <code>-mbranch-protection=standard</code> in GCC 9.1.0+ or
+Clang 10+ on Linux/AArch64 or for <code>/guard:signret</code> in Visual Studio
+2019+ on Windows/ARM64. The resulting build can be run on both machines with and
+without support for branch protection in hardware. Branch Protection is fully
+supported for the Linux/AArch64 target and only partially supported for the
+Windows/ARM64 targets.</p>
 <h3 id="building-on-32-bit-arm">Building on 32-bit ARM</h3>
 <p>This is not recommended. Instead, see the section on <a
 href="#cross-compiling">Cross-compiling</a>.</p>
@@ -1814,6 +1817,14 @@ Using ccache can radically speed up compilation of native code if you
 often rebuild the same sources. Your mileage may vary however, so we
 recommend evaluating it for yourself. To enable it, make sure it's on
 the path and configure with <code>--enable-ccache</code>.</p>
+<h3 id="sccache">Sccache</h3>
+<p>The JDK build supports building with sccache when using gcc, clang, or
+Microsoft toolchains.  To enable it, make sure the sccache binary is on the path
+(or specify the path to the binary using the <code>SCCACHE</code> argument to
+the configure script) and configure with <code>--enable-sccache</code>.  To
+optionally specify where sccache stores its cache files, use
+<code>--with-sccache-dir</code>.  Precompiled headers are disabled when sccache
+is enabled.</p>
 <h3 id="precompiled-headers">Precompiled Headers</h3>
 <p>By default, the Hotspot build uses pre-compiled headers (PCH) on the
 toolchains were it is properly supported (clang, gcc, and Visual

doc/building.md

@@ -171,9 +171,11 @@ possible to use [cross-compiling](#cross-compiling).
 
 In order to use Branch Protection features in the VM,
 `--enable-branch-protection` must be used. This option requires C++ compiler
-support (GCC 9.1.0+ or Clang 10+). The resulting build can be run on both
-machines with and without support for branch protection in hardware. Branch
-Protection is only supported for Linux targets.
+support for `-mbranch-protection=standard` in GCC 9.1.0+ or Clang 10+ on
+Linux/AArch64 or for `/guard:signret` in Visual Studio 2019+ on Windows/ARM64.
+The resulting build can be run on both machines with and without support for
+branch protection in hardware. Branch Protection is fully supported for the
+Linux/AArch64 target and only partially supported for the Windows/ARM64 target.
 
 ### Building on 32-bit ARM
 
@@ -1555,6 +1557,15 @@ the same sources. Your mileage may vary however, so we recommend evaluating it
 for yourself. To enable it, make sure it's on the path and configure with
 `--enable-ccache`.
 
+### Sccache
+
+The JDK build supports building with sccache when using gcc, clang, or Microsoft
+toolchains.  To enable it, make sure the sccache binary is on the path (or
+specify the path to the binary using the `SCCACHE` argument to the configure
+script) and configure with `--enable-sccache`.  To optionally specify where
+sccache stores its cache files, use `--with-sccache-dir`.  Precompiled headers
+are disabled when sccache is enabled.
+
 ### Precompiled Headers
 
 By default, the Hotspot build uses pre-compiled headers (PCH) on the toolchains

doc/testing.html

@@ -424,11 +424,29 @@ use the <code>--with-jcov-modules</code> arguments to
 <p>For more fine-grained control, you can pass arbitrary filters to JCov
 using <code>--with-jcov-filters</code>, and you can specify a specific
 JDK to instrument using <code>--with-jcov-input-jdk</code>.</p>
+<p>The resulting coverage is written into
+<code>build/$BUILD/test-results/jcov-output/result.xml</code>.</p>
 <p>The JCov report is stored in
 <code>build/$BUILD/test-results/jcov-output/report</code>.</p>
 <p>Please note that running with JCov reporting can be very memory
 intensive.</p>
-<h4 id="jcov_diff_changeset">JCOV_DIFF_CHANGESET</h4>
+<h5 id="jcov-scales">JCov scales</h5>
+<p>JCov scales make it possible to record which tests cover each part of
+the instrumented code. To collect coverage with scales, set
+<code>JCOV_SCALES=true</code>, for example:</p>
+<pre><code>$ make jcov-test TEST=jdk_lang TEST_OPTS=&quot;JCOV_SCALES=true&quot;</code></pre>
+<p>The resulting coverage data contains the association between covered
+code and the tests that covered it. A corresponding
+<code>testlist.txt</code> file, which contains the test names, is
+generated in the same directory.</p>
+<p>The JCov report displays the names of the tests that cover each
+class.</p>
+<p>Collecting coverage scales forces jtreg tests to be run in
+<code>othervm</code> mode, which takes longer than ordinary JCov
+collection. The coverage data is also larger because it includes scale
+information, and the generated report is larger because it includes test
+names.</p>
+<h5 id="jcov_diff_changeset">JCOV_DIFF_CHANGESET</h5>
 <p>While collecting code coverage with JCov, it is also possible to find
 coverage for only recently changed code. JCOV_DIFF_CHANGESET specifies a
 source revision. A textual report will be generated showing coverage of

doc/testing.md

@@ -353,11 +353,33 @@ For more fine-grained control, you can pass arbitrary filters to JCov using
 `--with-jcov-filters`, and you can specify a specific JDK to instrument
 using `--with-jcov-input-jdk`.
 
+The resulting coverage is written into
+`build/$BUILD/test-results/jcov-output/result.xml`.
+
 The JCov report is stored in `build/$BUILD/test-results/jcov-output/report`.
 
 Please note that running with JCov reporting can be very memory intensive.
 
-#### JCOV_DIFF_CHANGESET
+##### JCov scales
+
+JCov scales make it possible to record which tests cover each part of the
+instrumented code. To collect coverage with scales, set `JCOV_SCALES=true`,
+for example:
+
+    $ make jcov-test TEST=jdk_lang TEST_OPTS="JCOV_SCALES=true"
+
+The resulting coverage data contains the association between covered code and
+the tests that covered it. A corresponding `testlist.txt` file, which contains
+the test names, is generated in the same directory.
+
+The JCov report displays the names of the tests that cover each class.
+
+Collecting coverage scales forces jtreg tests to be run in `othervm` mode,
+which takes longer than ordinary JCov collection. The coverage data is also
+larger because it includes scale information, and the generated report is
+larger because it includes test names.
+
+##### JCOV_DIFF_CHANGESET
 
 While collecting code coverage with JCov, it is also possible to find coverage
 for only recently changed code. JCOV_DIFF_CHANGESET specifies a source

make/Main.gmk

@@ -751,6 +751,18 @@ $(eval $(call SetupTarget, test-image-lib, \
     DEPS := build-test-lib, \
 ))
 
+$(eval $(call SetupTarget, build-hotspot-compiler-testlibrary, \
+    MAKEFILE := hotspot/test/BuildCompilerTestlibrary, \
+    TARGET := build-hotspot-compiler-testlibrary, \
+    DEPS := exploded-image build-test-lib, \
+))
+
+$(eval $(call SetupTarget, test-image-hotspot-compiler-testlibrary, \
+    MAKEFILE := hotspot/test/BuildCompilerTestlibrary, \
+    TARGET := test-image-hotspot-compiler-testlibrary, \
+    DEPS := build-hotspot-compiler-testlibrary, \
+))
+
 $(eval $(call SetupTarget, build-test-setup-aot, \
     MAKEFILE := test/BuildTestSetupAOT, \
     DEPS := interim-langtools exploded-image, \
@@ -1303,7 +1315,7 @@ all-docs-bundles: docs-jdk-bundles docs-javase-bundles docs-reference-bundles
 test-image: prepare-test-image test-image-jdk-jtreg-native \
     test-image-demos-jdk test-image-libtest-jtreg-native \
     test-image-lib test-image-lib-native \
-    test-image-setup-aot
+    test-image-setup-aot test-image-hotspot-compiler-testlibrary
 
 ifneq ($(JVM_TEST_IMAGE_TARGETS), )
   # If JVM_TEST_IMAGE_TARGETS is externally defined, use it instead of the
@@ -1420,6 +1432,9 @@ clean: $(CLEAN_DIR_TARGETS)
 clean-docs:
 	$(call CleanDocs)
 
+clean-microbenchmark:
+	$(call CleanMicrobenchmark)
+
 clean-compile-commands:
 	$(call CleanMakeSupportDir,compile-commands)
 
@@ -1468,7 +1483,8 @@ dist-clean: clean
 	)
 	$(ECHO) Cleaned everything, you will have to re-run configure.
 
-ALL_TARGETS += clean clean-docs clean-compile-commands dist-clean $(CLEAN_DIR_TARGETS) \
+ALL_TARGETS += clean clean-docs clean-microbenchmark clean-compile-commands \
+    dist-clean $(CLEAN_DIR_TARGETS) \
     $(CLEAN_SUPPORT_DIR_TARGETS) $(CLEAN_TEST_TARGETS) $(CLEAN_PHASE_TARGETS) \
     $(CLEAN_MODULE_TARGETS) $(CLEAN_MODULE_PHASE_TARGETS)
 

make/MainSupport.gmk

@@ -65,6 +65,14 @@ define CleanDocs
 	@$(ECHO) " done"
 endef
 
+define CleanMicrobenchmark
+	@$(PRINTF) "Cleaning microbenchmark build artifacts ..."
+	@$(ECHO) "" $(LOG_DEBUG)
+	$(RM) -r $(SUPPORT_OUTPUTDIR)/test/micro
+	$(RM) -r $(TEST_IMAGE_DIR)/micro
+	@$(ECHO) " done"
+endef
+
 # Cleans the dir given as $1
 define CleanDir
 	@$(PRINTF) "Cleaning %s build artifacts ..." "$(strip $1)"

make/RunTests.gmk

@@ -45,7 +45,7 @@ ifneq ($(TEST_VM_OPTS), )
 endif
 
 $(eval $(call ParseKeywordVariable, TEST_OPTS, \
-    SINGLE_KEYWORDS := JOBS TIMEOUT_FACTOR JCOV JCOV_DIFF_CHANGESET AOT_JDK, \
+    SINGLE_KEYWORDS := JOBS TIMEOUT_FACTOR JCOV JCOV_DIFF_CHANGESET JCOV_SCALES AOT_JDK, \
     STRING_KEYWORDS := VM_OPTIONS JAVA_OPTIONS, \
 ))
 
@@ -121,9 +121,21 @@ ifeq ($(TEST_OPTS_JCOV), true)
   JCOV_SUPPORT_DIR := $(TEST_SUPPORT_DIR)/jcov-support
   JCOV_GRABBER_LOG := $(JCOV_OUTPUT_DIR)/grabber.log
   JCOV_RESULT_FILE := $(JCOV_OUTPUT_DIR)/result.xml
+  JCOV_TESTLIST := $(JCOV_OUTPUT_DIR)/testlist.txt
   JCOV_REPORT := $(JCOV_OUTPUT_DIR)/report
+  JCOV_GRABBER_OPTIONS ?=
+  JCOV_REPGEN_OPTIONS ?=
+  TEST_OPTS_JCOV_SCALES ?= false
   JCOV_MEM_OPTIONS := -Xms64m -Xmx4g
 
+  ifeq ($(TEST_OPTS_JCOV_SCALES), true)
+    JCOV_GRABBER_OPTIONS += -scale -mergebyname -outTestList $(JCOV_TESTLIST)
+    TEST_JOBS := 1
+    JTREG_TEST_MODE := othervm
+    JTREG_VM_OPTIONS += -Djcov.extension=com.sun.tdk.jcov.runtime.TestNameDecorator
+    JCOV_REPGEN_OPTIONS += -tests $(JCOV_TESTLIST)
+  endif
+
   # Replace our normal test JDK with the JCov image.
   JDK_UNDER_TEST := $(JCOV_IMAGE_DIR)
 
@@ -1054,9 +1066,11 @@ define SetupRunJtregTestBody
   $1_JTREG_BASIC_OPTIONS += -timeoutFactor:$$(JTREG_TIMEOUT_FACTOR)
 
   clean-outputdirs-$1:
-	$$(call LogWarn, Clean up dirs for $1)
-	$$(RM) -r $$($1_TEST_SUPPORT_DIR)
-	$$(RM) -r $$($1_TEST_RESULTS_DIR)
+        ifeq ($(JTREG_STATUS),)
+	  $$(call LogWarn, Clean up dirs for $1)
+	  $$(RM) -r $$($1_TEST_SUPPORT_DIR)
+	  $$(RM) -r $$($1_TEST_RESULTS_DIR)
+        endif
 
   $1_COMMAND_LINE := \
       $$(JTREG_JAVA) $$($1_JTREG_LAUNCHER_OPTIONS) \
@@ -1412,6 +1426,7 @@ ifeq ($(TEST_OPTS_JCOV), true)
 	fi
 	$(JAVA) $(JCOV_VM_OPTS) -jar $(JCOV_HOME)/lib/jcov.jar Grabber -v -t \
 	    $(JCOV_IMAGE_DIR)/template.xml -o $(JCOV_RESULT_FILE) \
+	    $(JCOV_GRABBER_OPTIONS) \
 	    1>$(JCOV_GRABBER_LOG) 2>&1 &
 
   jcov-start-grabber: jcov-do-start-grabber
@@ -1439,6 +1454,7 @@ ifeq ($(TEST_OPTS_JCOV), true)
 	    `$(ECHO) $(TOPDIR)/src/*/share/classes/ | $(TR) ' ' ':'` -fmt html \
 	    $(JCOV_MODULES_FILTER) $(JCOV_FILTERS) \
 	    -mainReportTitle "$(JCOV_REPORT_TITLE)" \
+	    $(JCOV_REPGEN_OPTIONS) \
 	    -o $(JCOV_REPORT) $(JCOV_RESULT_FILE))
 
   TARGETS += jcov-do-start-grabber jcov-start-grabber jcov-stop-grabber \

make/autoconf/build-performance.m4

@@ -252,6 +252,109 @@ AC_DEFUN([BPERF_SETUP_CCACHE_USAGE],
   fi
 ])
 
+AC_DEFUN([BPERF_SETUP_SCCACHE],
+[
+  # Check if sccache is available
+  SCCACHE_AVAILABLE=true
+
+  UTIL_LOOKUP_TOOLCHAIN_PROGS(SCCACHE, sccache)
+
+  AC_MSG_CHECKING([if sccache is available])
+  if test "x$TOOLCHAIN_TYPE" != "xgcc" && test "x$TOOLCHAIN_TYPE" != "xclang" && \
+      test "x$TOOLCHAIN_TYPE" != "xmicrosoft"; then
+    AC_MSG_RESULT([no, not supported for toolchain type $TOOLCHAIN_TYPE])
+    SCCACHE_AVAILABLE=false
+  elif test "x$SCCACHE" = "x"; then
+    AC_MSG_RESULT([no, sccache binary missing or not executable])
+    SCCACHE_AVAILABLE=false
+  else
+    AC_MSG_RESULT([yes])
+  fi
+
+  SCCACHE_STATUS=""
+  UTIL_ARG_ENABLE(NAME: sccache, DEFAULT: false, AVAILABLE: $SCCACHE_AVAILABLE,
+      DESC: [enable using sccache to speed up recompilations],
+      CHECKING_MSG: [if sccache is enabled],
+      IF_ENABLED: [
+        if test "x$CCACHE" != x; then
+          AC_MSG_ERROR([Cannot enable both ccache and sccache])
+        fi
+        # Versions of sccache before 0.10.0 can restore stale or incorrect
+        # dependency files for cached C/C++ compilations, breaking our build.
+        SCCACHE_VERSION=[`$SCCACHE --version | head -n1 | $CUT -d " " -f 2 | $TR -d '\r'`]
+        if test "x$SCCACHE_VERSION" = x; then
+          AC_MSG_ERROR([Could not determine sccache version])
+        fi
+        HAS_BAD_SCCACHE=[`$ECHO $SCCACHE_VERSION | \
+            $GREP -e '^0\.[0-9]\.' -e '^0\.[0-9]$'`]
+        if test "x$HAS_BAD_SCCACHE" != "x"; then
+          AC_MSG_ERROR([[sccache 0.10.0 or later is required, found $SCCACHE_VERSION]])
+        fi
+        SCCACHE_STATUS="Active ($SCCACHE_VERSION)"
+      ],
+      IF_DISABLED: [
+        SCCACHE=""
+      ])
+  AC_SUBST(SCCACHE)
+
+  AC_ARG_WITH([sccache-dir],
+      [AS_HELP_STRING([--with-sccache-dir],
+      [where to store sccache files @<:@~/.cache/sccache@:>@])])
+
+  if test "x$with_sccache_dir" != x; then
+    SCCACHE_DIR="$with_sccache_dir"
+    SCCACHE_DIR_FOR_SCCACHE="$SCCACHE_DIR"
+
+    # Ideally, we'd use `UTIL_FIXUP_PATH()`, but it expects the supplied path to
+    # already exist, which might not be true for the sccache directory during
+    # the configure step.  As a workaround, we manually invoke fixpath.sh.
+    if test "x$OPENJDK_BUILD_OS" = "xwindows"; then
+      SCCACHE_DIR_FOR_SCCACHE=`$FIXPATH_BASE -m print "$SCCACHE_DIR_FOR_SCCACHE"`
+    fi
+
+    SET_SCCACHE_DIR="SCCACHE_DIR=$SCCACHE_DIR_FOR_SCCACHE"
+    if test "x$SCCACHE" = x; then
+      AC_MSG_WARN([--with-sccache-dir has no meaning when sccache is not enabled])
+    fi
+  fi
+
+  if test "x$SCCACHE" != x; then
+    BPERF_SETUP_SCCACHE_USAGE
+  fi
+])
+
+AC_DEFUN([BPERF_SETUP_SCCACHE_USAGE],
+[
+  if test "x$SCCACHE" != x; then
+    if test "x$USE_PRECOMPILED_HEADER" = "xtrue"; then
+      if test "x$PRECOMPILED_HEADERS_EXPLICITLY_SET" = "xtrue"; then
+        AC_MSG_ERROR([Cannot use sccache with precompiled headers. Use --disable-precompiled-headers.])
+      else
+        AC_MSG_NOTICE([Disabling precompiled headers because sccache is enabled])
+        USE_PRECOMPILED_HEADER=false
+      fi
+    fi
+
+    # On Windows, the sccache binary must be launched through fixpath and the
+    # compiler argument passed to sccache must be the actual compiler
+    # (gcc/clang/cl) and not another fixpath invocation, otherwise sccache will
+    # try to execute fixpath as the compiler.
+    [ if [[ "$OPENJDK_BUILD_OS" = "windows" && "$SCCACHE" =~ ^"$FIXPATH " ]]; then ]
+      [ if [[ "$CC" =~ ^"$FIXPATH " ]]; then ]
+        CC="${CC#"$FIXPATH "}"
+      [ fi ]
+      [ if [[ "$CXX" =~ ^"$FIXPATH " ]]; then ]
+        CXX="${CXX#"$FIXPATH "}"
+      [ fi ]
+    [ fi ]
+
+    if test "x$SET_SCCACHE_DIR" != x; then
+      SCCACHE="$SET_SCCACHE_DIR $SCCACHE"
+      mkdir -p "$SCCACHE_DIR" > /dev/null 2>&1
+    fi
+  fi
+])
+
 ################################################################################
 #
 # Runs icecc-create-env once and prints the error if it fails
@@ -372,7 +475,13 @@ AC_DEFUN_ONCE([BPERF_SETUP_PRECOMPILED_HEADERS],
 
   UTIL_ARG_ENABLE(NAME: precompiled-headers, DEFAULT: auto,
       RESULT: USE_PRECOMPILED_HEADER, AVAILABLE: $PRECOMPILED_HEADERS_AVAILABLE,
-      DESC: [enable using precompiled headers when compiling C++])
+      DESC: [enable using precompiled headers when compiling C++],
+      IF_GIVEN: [
+        PRECOMPILED_HEADERS_EXPLICITLY_SET=true
+      ],
+      IF_NOT_GIVEN: [
+        PRECOMPILED_HEADERS_EXPLICITLY_SET=false
+      ])
   AC_SUBST(USE_PRECOMPILED_HEADER)
 ])
 

make/autoconf/configure.ac

@@ -293,6 +293,9 @@ BPERF_SETUP_PRECOMPILED_HEADERS
 # Setup use of ccache, if available
 BPERF_SETUP_CCACHE
 
+# Setup use of sccache, if available
+BPERF_SETUP_SCCACHE
+
 ################################################################################
 #
 # And now the finish...

make/autoconf/flags-cflags.m4

@@ -945,11 +945,21 @@ AC_DEFUN_ONCE([FLAGS_SETUP_BRANCH_PROTECTION],
 [
   # Is branch protection available?
   BRANCH_PROTECTION_AVAILABLE=false
-  BRANCH_PROTECTION_FLAG="-mbranch-protection=standard"
+  BRANCH_PROTECTION_CFLAG=""
+  BRANCH_PROTECTION_ASFLAG=""
 
   if test "x$OPENJDK_TARGET_CPU" = xaarch64; then
     if test "x$TOOLCHAIN_TYPE" = xgcc || test "x$TOOLCHAIN_TYPE" = xclang; then
-      FLAGS_COMPILER_CHECK_ARGUMENTS(ARGUMENT: [$BRANCH_PROTECTION_FLAG],
+      BRANCH_PROTECTION_CFLAG="-mbranch-protection=standard"
+      # The GCC/Clang assembler accepts the same flag as the compiler.
+      BRANCH_PROTECTION_ASFLAG="$BRANCH_PROTECTION_CFLAG"
+      FLAGS_COMPILER_CHECK_ARGUMENTS(ARGUMENT: [$BRANCH_PROTECTION_CFLAG],
+          IF_TRUE: [BRANCH_PROTECTION_AVAILABLE=true])
+    elif test "x$TOOLCHAIN_TYPE" = xmicrosoft; then
+      BRANCH_PROTECTION_CFLAG="/guard:signret"
+      # MSVC's assembler does not support branch protection flags, so
+      # BRANCH_PROTECTION_ASFLAG is intentionally left empty.
+      FLAGS_COMPILER_CHECK_ARGUMENTS(ARGUMENT: [$BRANCH_PROTECTION_CFLAG],
           IF_TRUE: [BRANCH_PROTECTION_AVAILABLE=true])
     fi
   fi
@@ -957,5 +967,5 @@ AC_DEFUN_ONCE([FLAGS_SETUP_BRANCH_PROTECTION],
   UTIL_ARG_ENABLE(NAME: branch-protection, DEFAULT: false,
       RESULT: BRANCH_PROTECTION_ENABLED, AVAILABLE: $BRANCH_PROTECTION_AVAILABLE,
       DESC: [enable branch protection when compiling C/C++],
-      IF_ENABLED: [BRANCH_PROTECTION_CFLAGS=$BRANCH_PROTECTION_FLAG])
+      IF_ENABLED: [BRANCH_PROTECTION_CFLAGS=$BRANCH_PROTECTION_CFLAG])
 ])

make/autoconf/flags-other.m4

@@ -211,7 +211,7 @@ AC_DEFUN([FLAGS_SETUP_ASFLAGS_CPU_DEP],
   fi
 
   if test "x$BRANCH_PROTECTION_ENABLED" = "xtrue"; then
-    $2JVM_ASFLAGS="${$2JVM_ASFLAGS} $BRANCH_PROTECTION_FLAG"
+    $2JVM_ASFLAGS="${$2JVM_ASFLAGS} $BRANCH_PROTECTION_ASFLAG"
   fi
 
   AC_SUBST($2JVM_ASFLAGS)

make/autoconf/help.m4

@@ -335,6 +335,9 @@ AC_DEFUN_ONCE([HELP_PRINT_SUMMARY_AND_WARNINGS],
   if test "x$CCACHE_STATUS" != "x"; then
     $ECHO "* ccache status:  $CCACHE_STATUS"
   fi
+  if test "x$SCCACHE_STATUS" != "x"; then
+    $ECHO "* sccache status: $SCCACHE_STATUS"
+  fi
   $ECHO ""
 
   if test "x$BUILDING_MULTIPLE_JVM_VARIANTS" = "xtrue"; then

make/autoconf/platform.m4

@@ -660,7 +660,19 @@ AC_DEFUN([PLATFORM_CHECK_DEPRECATION],
 [
   AC_ARG_ENABLE(deprecated-ports, [AS_HELP_STRING([--enable-deprecated-ports@<:@=yes/no@:>@],
       [Suppress the error when configuring for a deprecated port @<:@no@:>@])])
-  # There are no deprecated ports. Implement the deprecation warnings here.
+  if test "x$OPENJDK_TARGET_OS" = xmacosx && test "x$OPENJDK_TARGET_CPU" = xx86_64; then
+    # Unfortunately, variants have not been parsed yet, so we have to check the configure option
+    # directly. Allow only the directly specified Zero variant, treat any other mix as containing
+    # something non-Zero.
+    if test "x$with_jvm_variants" != xzero; then
+      if test "x$enable_deprecated_ports" = "xyes"; then
+        AC_MSG_WARN([The macOS/x64 port is deprecated and may be removed in a future release.])
+      else
+        AC_MSG_ERROR(m4_normalize([The macOS/x64 port is deprecated and may be removed in a future release.
+          Use --enable-deprecated-ports to suppress this error.]))
+      fi
+    fi
+  fi
 ])
 
 AC_DEFUN_ONCE([PLATFORM_SETUP_OPENJDK_BUILD_OS_VERSION],

make/autoconf/spec.gmk.template

@@ -545,7 +545,7 @@ ADLC_LANGSTD_CXXFLAGS := @ADLC_LANGSTD_CXXFLAGS@
 ADLC_LDFLAGS := @ADLC_LDFLAGS@
 
 # Tools that potentially need to be cross compilation aware.
-CC := @CCACHE@ @ICECC@ @CC@
+CC := @SCCACHE@ @CCACHE@ @ICECC@ @CC@
 
 # CFLAGS used to compile the jdk native libraries (C-code)
 CFLAGS_JDKLIB := @CFLAGS_JDKLIB@
@@ -571,7 +571,7 @@ EXTRA_CXXFLAGS := @EXTRA_CXXFLAGS@
 EXTRA_LDFLAGS := @EXTRA_LDFLAGS@
 EXTRA_ASFLAGS := @EXTRA_ASFLAGS@
 
-CXX := @CCACHE@ @ICECC@ @CXX@
+CXX := @SCCACHE@ @CCACHE@ @ICECC@ @CXX@
 
 CPP := @CPP@
 
@@ -740,6 +740,7 @@ RCFLAGS := @RCFLAGS@
 AWK := @AWK@
 BASENAME := @BASENAME@
 CAT := @CAT@
+SCCACHE := @SCCACHE@
 CCACHE := @CCACHE@
 # CD is going away, but remains to cater for legacy makefiles.
 CD := cd

make/common/JavaCompilation.gmk

@@ -148,8 +148,8 @@ endef
 #   BIN store classes here
 #   MODULE Name of module being compiled. If set, classes are put in BIN/MODULE.
 #   CLASSPATH a list of additional entries to set as classpath to javac
-#   INCLUDES myapp.foo means will only compile java files in myapp.foo or any of its sub-packages.
-#   EXCLUDES myapp.foo means will do not compile java files in myapp.foo or any of its sub-packages.
+#   INCLUDES myapp/foo means will only compile java files in myapp.foo or any of its sub-packages.
+#   EXCLUDES myapp/foo means will do not compile java files in myapp.foo or any of its sub-packages.
 #   COPY .prp means copy all prp files to the corresponding package in BIN.
 #   COPY_FILES myapp/foo/setting.txt means copy this file over to the package myapp/foo
 #   CLEAN .properties means copy and clean all properties file to the corresponding package in BIN.

make/conf/jib-profiles.js

@@ -416,6 +416,7 @@ var getJibProfilesProfiles = function (input, common, data) {
                 "--with-zlib=system",
                 "--with-macosx-version-max=11.00.00",
                 "--enable-compatible-cds-alignment",
+                "--enable-deprecated-ports",
                 // Use system SetFile instead of the one in the devkit as the
                 // devkit one may not work on Catalina.
                 "SETFILE=/usr/bin/SetFile"
@@ -1192,8 +1193,8 @@ var getJibProfilesDependencies = function (input, common) {
             server: "jpg",
             product: "jcov",
             version: "3.0",
-            build_number: "6",
-            file: "bundles/jcov-3.0+6.zip",
+            build_number: "9",
+            file: "bundles/jcov-3.0+9.zip",
             environment_name: "JCOV_HOME",
         },
 

make/hotspot/lib/CompileJvm.gmk

@@ -285,13 +285,6 @@ ifeq ($(call isTargetOs, windows), true)
   $(BUILD_LIBJVM_TARGET): $(WIN_EXPORT_FILE)
 endif
 
-# Always recompile abstract_vm_version.cpp if libjvm needs to be relinked. This ensures
-# that the internal vm version is updated as it relies on __DATE__ and __TIME__
-# macros.
-ABSTRACT_VM_VERSION_OBJ := $(JVM_OUTPUTDIR)/objs/abstract_vm_version$(OBJ_SUFFIX)
-$(ABSTRACT_VM_VERSION_OBJ): $(filter-out $(ABSTRACT_VM_VERSION_OBJ), \
-    $(BUILD_LIBJVM_TARGET_DEPS))
-
 ifneq ($(GENERATE_COMPILE_COMMANDS_ONLY), true)
   ifeq ($(call isTargetOs, windows), true)
     # It doesn't matter which jvm.lib file gets exported, but we need

make/hotspot/test/BuildCompilerTestlibrary.gmk

@@ -0,0 +1,76 @@
+#
+# Copyright (c) 2026, Oracle and/or its affiliates. All rights reserved.
+# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+#
+# This code is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 2 only, as
+# published by the Free Software Foundation.  Oracle designates this
+# particular file as subject to the "Classpath" exception as provided
+# by Oracle in the LICENSE file that accompanied this code.
+#
+# This code is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# version 2 for more details (a copy is included in the LICENSE file that
+# accompanied this code).
+#
+# You should have received a copy of the GNU General Public License version
+# 2 along with this work; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+# or visit www.oracle.com if you need additional information or have any
+# questions.
+#
+
+include MakeFileStart.gmk
+
+################################################################################
+# This file builds the Hotspot compiler testlibrary.
+################################################################################
+
+include CopyFiles.gmk
+include JavaCompilation.gmk
+
+###############################################################################
+
+COMPILER_TESTLIBRARY_BASEDIR := $(TOPDIR)/test/hotspot/jtreg/compiler/lib
+COMPILER_TESTLIBRARY_SUPPORT := $(SUPPORT_OUTPUTDIR)/test/compiler-testlibrary
+COMPILER_TESTLIBRARY_JAR := $(COMPILER_TESTLIBRARY_SUPPORT)/compiler-testlibrary.jar
+TEST_LIB_SUPPORT := $(SUPPORT_OUTPUTDIR)/test/lib
+WB_CP := $(TEST_LIB_SUPPORT)/wb_classes
+TEST_LIB_CP := $(TEST_LIB_SUPPORT)/test-lib_classes
+
+$(eval $(call SetupJavaCompilation, BUILD_COMPILER_TESTLIBRARY, \
+    TARGET_RELEASE := $(TARGET_RELEASE_NEWJDK_UPGRADED), \
+    SRC := $(COMPILER_TESTLIBRARY_BASEDIR), \
+    BIN := $(COMPILER_TESTLIBRARY_SUPPORT)/classes, \
+    JAR := $(COMPILER_TESTLIBRARY_JAR), \
+    JAVAC_FLAGS := -cp $(WB_CP) \
+        -cp $(TEST_LIB_CP) \
+        --add-exports java.base/jdk.internal.math=ALL-UNNAMED, \
+))
+
+TARGETS += $(BUILD_COMPILER_TESTLIBRARY)
+
+build-hotspot-compiler-testlibrary: $(TARGETS)
+
+################################################################################
+# Targets for building test-image.
+################################################################################
+
+# Copy to hotspot jtreg test image
+$(eval $(call SetupCopyFiles, COPY_COMPILER_TESTLIBRARY, \
+    DEST := $(TEST_IMAGE_DIR)/compiler-testlibrary, \
+    FILES := $(COMPILER_TESTLIBRARY_JAR), \
+))
+
+IMAGE_TARGETS += $(COPY_COMPILER_TESTLIBRARY)
+
+test-image-hotspot-compiler-testlibrary: $(IMAGE_TARGETS)
+
+.PHONY: build-hotspot-compiler-testlibrary test-image-hotspot-compiler-testlibrary
+
+################################################################################
+
+include MakeFileEnd.gmk

make/test/BuildMicrobenchmark.gmk

@@ -77,44 +77,74 @@ MICROBENCHMARK_MANIFEST := Build: $(FULL_VERSION)\n\
 # jmh uses annotation processors to generate the benchmark jar and thus
 # requires the use of -processor option during benchmark compilation.
 
+MICROBENCHMARK_PREVIEW_PACKAGES := \
+    org/openjdk/bench/valhalla \
+    org/openjdk/bench/java/lang/stable
+
+MICROBENCHMARK_JAVAC_FLAGS := \
+    --add-exports java.base/jdk.internal.classfile.components=ALL-UNNAMED \
+    --add-exports java.base/jdk.internal.classfile.impl=ALL-UNNAMED \
+    --add-exports java.base/jdk.internal.event=ALL-UNNAMED \
+    --add-exports java.base/jdk.internal.foreign=ALL-UNNAMED \
+    --add-exports java.base/jdk.internal.jimage=ALL-UNNAMED \
+    --add-exports java.base/jdk.internal.misc=ALL-UNNAMED \
+    --add-exports java.base/jdk.internal.util=ALL-UNNAMED \
+    --add-exports java.base/jdk.internal.value=ALL-UNNAMED \
+    --add-exports java.base/jdk.internal.vm=ALL-UNNAMED \
+    --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED \
+    --add-exports java.base/sun.invoke.util=ALL-UNNAMED \
+    --add-exports java.base/sun.security.util=ALL-UNNAMED \
+    --add-exports java.base/sun.security.util.math=ALL-UNNAMED \
+    --add-exports java.base/sun.security.util.math.intpoly=ALL-UNNAMED \
+    -XDsuppressNotes \
+    -processor org.openjdk.jmh.generators.BenchmarkProcessor \
+    -s $(MICROBENCHMARK_GENSRC)
+
+# For code built with --enable-preview we need to patch java.base
+# to include preview classes not found in interim javac
+MICROBENCHMARK_JAVAC_PREVIEW_FLAGS := \
+    $(MICROBENCHMARK_JAVAC_FLAGS) \
+    --enable-preview \
+    --patch-module java.base=$(SUPPORT_OUTPUTDIR)/preview/java.base
+
+MICROBENCHMARK_JAVA_FLAGS := \
+    --add-exports java.base/jdk.internal.vm=ALL-UNNAMED \
+    --add-modules jdk.unsupported \
+    --limit-modules java.management
+
+MICROBENCHMARK_DISABLED_WARNINGS := restricted this-escape processing \
+    rawtypes removal cast serial preview unchecked deprecation \
+    dangling-doc-comments
+
 # Build microbenchmark suite for the current JDK
-# Need to patch java.base to include preview classes not found in interim javac
+$(eval $(call SetupJavaCompilation, BUILD_JDK_MICROBENCHMARK_PREVIEW, \
+    TARGET_RELEASE := $(TARGET_RELEASE_NEWJDK_UPGRADED), \
+    SMALL_JAVA := false, \
+    CLASSPATH := $(JMH_COMPILE_JARS), \
+    INCLUDES := $(MICROBENCHMARK_PREVIEW_PACKAGES), \
+    CREATE_API_DIGEST := true, \
+    DISABLED_WARNINGS := $(MICROBENCHMARK_DISABLED_WARNINGS), \
+    SRC := $(MICROBENCHMARK_SRC), \
+    BIN := $(MICROBENCHMARK_CLASSES), \
+    JAVAC_FLAGS := $(MICROBENCHMARK_JAVAC_PREVIEW_FLAGS), \
+    JAVA_FLAGS := $(MICROBENCHMARK_JAVA_FLAGS), \
+))
+
 $(eval $(call SetupJavaCompilation, BUILD_JDK_MICROBENCHMARK, \
     TARGET_RELEASE := $(TARGET_RELEASE_NEWJDK_UPGRADED), \
     SMALL_JAVA := false, \
     CLASSPATH := $(JMH_COMPILE_JARS), \
     CREATE_API_DIGEST := true, \
-    DISABLED_WARNINGS := restricted this-escape processing rawtypes removal cast \
-        serial preview unchecked deprecation dangling-doc-comments, \
+    EXCLUDES := $(MICROBENCHMARK_PREVIEW_PACKAGES), \
+    DISABLED_WARNINGS := $(MICROBENCHMARK_DISABLED_WARNINGS), \
     SRC := $(MICROBENCHMARK_SRC), \
     BIN := $(MICROBENCHMARK_CLASSES), \
-    JAVAC_FLAGS := \
-        --add-exports java.base/jdk.internal.classfile.components=ALL-UNNAMED \
-        --add-exports java.base/jdk.internal.classfile.impl=ALL-UNNAMED \
-        --add-exports java.base/jdk.internal.event=ALL-UNNAMED \
-        --add-exports java.base/jdk.internal.foreign=ALL-UNNAMED \
-        --add-exports java.base/jdk.internal.jimage=ALL-UNNAMED \
-        --add-exports java.base/jdk.internal.misc=ALL-UNNAMED \
-        --add-exports java.base/jdk.internal.util=ALL-UNNAMED \
-        --add-exports java.base/jdk.internal.value=ALL-UNNAMED \
-        --add-exports java.base/jdk.internal.vm=ALL-UNNAMED \
-        --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED \
-        --add-exports java.base/sun.invoke.util=ALL-UNNAMED \
-        --add-exports java.base/sun.security.util=ALL-UNNAMED \
-        --add-exports java.base/sun.security.util.math=ALL-UNNAMED \
-        --add-exports java.base/sun.security.util.math.intpoly=ALL-UNNAMED \
-        --enable-preview \
-        --patch-module java.base=$(SUPPORT_OUTPUTDIR)/preview/java.base \
-        -XDsuppressNotes \
-        -processor org.openjdk.jmh.generators.BenchmarkProcessor \
-        -s $(MICROBENCHMARK_GENSRC), \
-    JAVA_FLAGS := \
-        --add-exports java.base/jdk.internal.vm=ALL-UNNAMED \
-        --add-modules jdk.unsupported \
-        --enable-preview \
-        --limit-modules java.management, \
+    JAVAC_FLAGS := $(MICROBENCHMARK_JAVAC_FLAGS), \
+    JAVA_FLAGS := $(MICROBENCHMARK_JAVA_FLAGS), \
 ))
 
+$(BUILD_JDK_MICROBENCHMARK_PREVIEW): $(JMH_COMPILE_JARS)
+
 $(BUILD_JDK_MICROBENCHMARK): $(JMH_COMPILE_JARS)
 
 # Unpacking dependencies for inclusion in the benchmark JARs
@@ -139,12 +169,16 @@ $(eval $(call SetupCopyFiles, COPY_JAXP_TEST_XML, \
 
 # Create benchmarks JAR file with benchmarks for both the old and new JDK
 $(eval $(call SetupJarArchive, BUILD_JDK_JAR, \
-    DEPENDENCIES := $(BUILD_JDK_MICROBENCHMARK) $(JMH_UNPACKED_JARS_DONE) $(COPY_JAXP_TEST_XML), \
+    DEPENDENCIES := $(BUILD_JDK_MICROBENCHMARK_PREVIEW) \
+        $(BUILD_JDK_MICROBENCHMARK) $(JMH_UNPACKED_JARS_DONE) \
+        $(COPY_JAXP_TEST_XML), \
     SRCS := $(MICROBENCHMARK_CLASSES) $(JMH_UNPACKED_DIR), \
     BIN := $(MICROBENCHMARK_JAR_BIN), \
     SUFFIXES := .*, \
     EXCLUDE_FILES := _the.BUILD_JDK_MICROBENCHMARK_batch \
-        _the.BUILD_JDK_MICROBENCHMARK.vardeps _unpacked.marker, \
+        _the.BUILD_JDK_MICROBENCHMARK.vardeps _unpacked.marker \
+        _the.BUILD_JDK_MICROBENCHMARK_PREVIEW_batch \
+        _the.BUILD_JDK_MICROBENCHMARK_PREVIEW.vardeps, \
     EXTRA_MANIFEST_ATTR := $(MICROBENCHMARK_MANIFEST), \
     JARMAIN := org.openjdk.jmh.Main, \
     JAR := $(MICROBENCHMARK_JAR), \

src/hotspot/cpu/aarch64/assembler_aarch64.hpp

@@ -457,17 +457,18 @@ class Address {
 
   Address(address target, relocInfo::relocType rtype = relocInfo::external_word_type);
 
-  Address(Register base, RegisterOrConstant index, extend ext = lsl()) {
+  Address(Register base, RegisterOrConstant index, extend ext = lsl(0)) {
     if (index.is_register()) {
       _mode = base_plus_offset_reg;
       new (&_nonliteral) Nonliteral(base, index.as_register(), 0, ext);
     } else {
       guarantee(ext.option() == ext::uxtx, "should be");
       assert(index.is_constant(), "should be");
+      assert(ext.shift() == 0, "must be");
       _mode = base_plus_offset;
       new (&_nonliteral) Nonliteral(base,
                                     noreg,
-                                    index.as_constant() << ext.shift());
+                                    index.as_constant());
     }
   }
 

src/hotspot/cpu/aarch64/bytes_aarch64.hpp

@@ -1,57 +0,0 @@
-/*
- * Copyright (c) 1997, 2022, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2014, Red Hat Inc. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#ifndef CPU_AARCH64_BYTES_AARCH64_HPP
-#define CPU_AARCH64_BYTES_AARCH64_HPP
-
-#include "memory/allStatic.hpp"
-#include "utilities/byteswap.hpp"
-
-class Bytes: AllStatic {
- public:
-  // Efficient reading and writing of unaligned unsigned data in platform-specific byte ordering
-  // (no special code is needed since x86 CPUs can access unaligned data)
-  static inline u2   get_native_u2(address p)         { return *(u2*)p; }
-  static inline u4   get_native_u4(address p)         { return *(u4*)p; }
-  static inline u8   get_native_u8(address p)         { return *(u8*)p; }
-
-  static inline void put_native_u2(address p, u2 x)   { *(u2*)p = x; }
-  static inline void put_native_u4(address p, u4 x)   { *(u4*)p = x; }
-  static inline void put_native_u8(address p, u8 x)   { *(u8*)p = x; }
-
-
-  // Efficient reading and writing of unaligned unsigned data in Java
-  // byte ordering (i.e. big-endian ordering). Byte-order reversal is
-  // needed since x86 CPUs use little-endian format.
-  static inline u2   get_Java_u2(address p)           { return byteswap(get_native_u2(p)); }
-  static inline u4   get_Java_u4(address p)           { return byteswap(get_native_u4(p)); }
-  static inline u8   get_Java_u8(address p)           { return byteswap(get_native_u8(p)); }
-
-  static inline void put_Java_u2(address p, u2 x)     { put_native_u2(p, byteswap(x)); }
-  static inline void put_Java_u4(address p, u4 x)     { put_native_u4(p, byteswap(x)); }
-  static inline void put_Java_u8(address p, u8 x)     { put_native_u8(p, byteswap(x)); }
-};
-
-#endif // CPU_AARCH64_BYTES_AARCH64_HPP

src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.cpp

@@ -182,10 +182,8 @@ void C2_MacroAssembler::fast_lock(Register obj, Register box, Register t1,
   // Finish fast lock unsuccessfully. MUST branch to with flag == NE
   Label slow_path;
 
-  if (UseObjectMonitorTable) {
-    // Clear cache in case fast locking succeeds or we need to take the slow-path.
-    str(zr, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
-  }
+  // Clear cache in case fast locking succeeds or we need to take the slow-path.
+  str(zr, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
 
   if (DiagnoseSyncOnValueBasedClasses != 0) {
     load_klass(t1, obj, rscratch2);
@@ -245,60 +243,55 @@ void C2_MacroAssembler::fast_lock(Register obj, Register box, Register t1,
     const ByteSize omc_monitor_offset = OMCache::monitor_offset();
     const ByteSize omc_obj_offset     = OMCache::obj_offset();
 
-    if (!UseObjectMonitorTable) {
-      assert(t1_monitor == t1_mark, "should be the same here");
-    } else {
-      const Register t1_hash = t1;
-      Label monitor_found;
+    const Register t1_hash = t1;
+    Label monitor_found;
 
-      // Save the mark, we might need it to extract the hash.
-      mov(t3, t1_mark);
+    // Save the mark, we might need it to extract the hash.
+    mov(t3, t1_mark);
 
-      // Look for the monitor in the current thread's object monitor cache (omc).
+    // Look for the monitor in the current thread's object monitor cache (omc).
 
-      ldr(t1_monitor, Address(rthread, thr_omc_offset + omc_monitor_offset));
-      ldr(t2, Address(rthread, thr_omc_offset + omc_obj_offset));
-      cmp(obj, t2);
-      br(Assembler::EQ, monitor_found);
+    ldr(t1_monitor, Address(rthread, thr_omc_offset + omc_monitor_offset));
+    ldr(t2, Address(rthread, thr_omc_offset + omc_obj_offset));
+    cmp(obj, t2);
+    br(Assembler::EQ, monitor_found);
 
-      // Look for the monitor in the table.
+    // Look for the monitor in the table.
 
-      // Get the hash code.
-      ubfx(t1_hash, t3, markWord::hash_shift, markWord::hash_bits);
+    // Get the hash code.
+    ubfx(t1_hash, t3, markWord::hash_shift, markWord::hash_bits);
 
-      // Get the table and calculate the bucket's address
-      lea(t3, ExternalAddress(ObjectMonitorTable::current_table_address()));
-      ldr(t3, Address(t3));
-      ldr(t2, Address(t3, ObjectMonitorTable::table_capacity_mask_offset()));
-      ands(t1_hash, t1_hash, t2);
-      ldr(t3, Address(t3, ObjectMonitorTable::table_buckets_offset()));
+    // Get the table and calculate the bucket's address
+    lea(t3, ExternalAddress(ObjectMonitorTable::current_table_address()));
+    ldr(t3, Address(t3));
+    ldr(t2, Address(t3, ObjectMonitorTable::table_capacity_mask_offset()));
+    ands(t1_hash, t1_hash, t2);
+    ldr(t3, Address(t3, ObjectMonitorTable::table_buckets_offset()));
 
-      // Read the monitor from the bucket.
-      ldr(t1_monitor, Address(t3, t1_hash, Address::lsl(LogBytesPerWord)));
+    // Read the monitor from the bucket.
+    ldr(t1_monitor, Address(t3, t1_hash, Address::lsl(LogBytesPerWord)));
 
-      // Check if the monitor in the bucket is special (empty, tombstone or removed).
-      cmp(t1_monitor, (unsigned char)ObjectMonitorTable::SpecialPointerValues::below_is_special);
-      br(Assembler::LO, slow_path);
+    // Check if the monitor in the bucket is special (empty, tombstone or removed).
+    cmp(t1_monitor, (unsigned char)ObjectMonitorTable::SpecialPointerValues::below_is_special);
+    br(Assembler::LO, slow_path);
 
-      // Check if object matches.
-      ldr(t3, Address(t1_monitor, ObjectMonitor::object_offset()));
-      BarrierSetAssembler* bs_asm = BarrierSet::barrier_set()->barrier_set_assembler();
-      bs_asm->try_peek_weak_handle_in_nmethod(this, t3, t3, t2, slow_path);
-      cmp(t3, obj);
-      br(Assembler::NE, slow_path);
+    // Check if object matches.
+    ldr(t3, Address(t1_monitor, ObjectMonitor::object_offset()));
+    BarrierSetAssembler* bs_asm = BarrierSet::barrier_set()->barrier_set_assembler();
+    bs_asm->try_peek_weak_handle_in_nmethod(this, t3, t3, t2, slow_path);
+    cmp(t3, obj);
+    br(Assembler::NE, slow_path);
 
-      // Store the monitor in the current thread's object monitor cache (omc).
-      str(t1_monitor, Address(rthread, thr_omc_offset + omc_monitor_offset));
-      str(obj, Address(rthread, thr_omc_offset + omc_obj_offset));
+    // Store the monitor in the current thread's object monitor cache (omc).
+    str(t1_monitor, Address(rthread, thr_omc_offset + omc_monitor_offset));
+    str(obj, Address(rthread, thr_omc_offset + omc_obj_offset));
 
-      bind(monitor_found);
-    }
+    bind(monitor_found);
 
     const Register t2_owner_addr = t2;
     const Register t3_owner = t3;
-    const ByteSize monitor_tag = in_ByteSize(UseObjectMonitorTable ? 0 : checked_cast<int>(markWord::monitor_value));
-    const Address owner_address(t1_monitor, ObjectMonitor::owner_offset() - monitor_tag);
-    const Address recursions_address(t1_monitor, ObjectMonitor::recursions_offset() - monitor_tag);
+    const Address owner_address(t1_monitor, ObjectMonitor::owner_offset());
+    const Address recursions_address(t1_monitor, ObjectMonitor::recursions_offset());
 
     Label monitor_locked;
 
@@ -318,10 +311,8 @@ void C2_MacroAssembler::fast_lock(Register obj, Register box, Register t1,
     increment(recursions_address, 1);
 
     bind(monitor_locked);
-    if (UseObjectMonitorTable) {
-      // Cache the monitor for unlock.
-      str(t1_monitor, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
-    }
+    // Cache the monitor for unlock.
+    str(t1_monitor, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
   }
 
   bind(locked);
@@ -388,7 +379,7 @@ void C2_MacroAssembler::fast_unlock(Register obj, Register box, Register t1,
     // Because we got here by popping (meaning we pushed in locked)
     // there will be no monitor in the box. So we need to push back the obj
     // so that the runtime can fix any potential anonymous owner.
-    tbnz(t1_mark, exact_log2(markWord::monitor_value), UseObjectMonitorTable ? push_and_slow_path : inflated);
+    tbnz(t1_mark, exact_log2(markWord::monitor_value), push_and_slow_path);
 
     // Try to unlock. Transition lock bits 0b00 => 0b01
     assert(oopDesc::mark_offset_in_bytes() == 0, "required to avoid lea");
@@ -430,17 +421,10 @@ void C2_MacroAssembler::fast_unlock(Register obj, Register box, Register t1,
 
     const Register t1_monitor = t1;
 
-    if (!UseObjectMonitorTable) {
-      assert(t1_monitor == t1_mark, "should be the same here");
-
-      // Untag the monitor.
-      add(t1_monitor, t1_mark, -(int)markWord::monitor_value);
-    } else {
-      ldr(t1_monitor, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
-      // null check with Flags == NE, no valid pointer below alignof(ObjectMonitor*)
-      cmp(t1_monitor, checked_cast<uint8_t>(alignof(ObjectMonitor*)));
-      br(Assembler::LO, slow_path);
-    }
+    ldr(t1_monitor, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
+    // null check with Flags == NE, no valid pointer below alignof(ObjectMonitor*)
+    cmp(t1_monitor, checked_cast<uint8_t>(alignof(ObjectMonitor*)));
+    br(Assembler::LO, slow_path);
 
     const Register t2_recursions = t2;
     Label not_recursive;
@@ -2452,7 +2436,8 @@ void C2_MacroAssembler::neon_reverse_bytes(FloatRegister dst, FloatRegister src,
 void C2_MacroAssembler::neon_rearrange_hsd(FloatRegister dst, FloatRegister src,
                                            FloatRegister shuffle, FloatRegister tmp,
                                            BasicType bt, bool isQ) {
-  assert_different_registers(dst, src, shuffle, tmp);
+  assert_different_registers(dst, src, tmp);
+  assert_different_registers(shuffle, tmp);
   SIMD_Arrangement size1 = isQ ? T16B : T8B;
   SIMD_Arrangement size2 = esize2arrangement((uint)type2aelembytes(bt), isQ);
 
@@ -3032,4 +3017,4 @@ void C2_MacroAssembler::sve_sdiv_short(FloatRegister dst_src1, FloatRegister src
   sve_sdiv(src1, S, ptrue, vtmp2);
   // Narrow the two INT result halves back to SHORT.
   sve_uzp1(dst_src1, H, vtmp1, src1);
-}
+}

src/hotspot/cpu/aarch64/downcallLinker_aarch64.cpp

@@ -308,7 +308,7 @@ void DowncallLinker::StubGenerator::generate() {
     // Restore cpu control state after JNI call
     __ restore_cpu_control_state_after_jni(rscratch1, tmp1);
 
-    __ mov(tmp1, _thread_in_native_trans);
+    __ mov(tmp1, _thread_in_vm);
     __ strw(tmp1, Address(rthread, JavaThread::thread_state_offset()));
 
     // Force this write out before the read below
@@ -388,5 +388,5 @@ void DowncallLinker::StubGenerator::generate() {
 
   //////////////////////////////////////////////////////////////////////////////
 
-  __ flush();
+  // Code will be copied. No ICache sync required.
 }

src/hotspot/cpu/aarch64/frame_aarch64.cpp

@@ -805,8 +805,8 @@ intptr_t* frame::repair_sender_sp(nmethod* nm, intptr_t* sp, intptr_t** saved_fp
 }
 
 bool frame::was_augmented_on_entry(int& real_size) const {
-  assert(is_compiled_frame(), "");
-  if (_cb->as_nmethod_or_null()->needs_stack_repair()) {
+  assert(_cb != nullptr && _cb->is_nmethod(), "");
+  if (_cb->as_nmethod()->needs_stack_repair()) {
     // The stack increment resides just below the saved FP on the stack and
     // records the total frame size excluding the two words for saving FP and LR
     // (see MacroAssembler::remove_frame).

src/hotspot/cpu/aarch64/gc/shared/barrierSetAssembler_aarch64.cpp

@@ -23,6 +23,7 @@
  */
 
 #include "classfile/classLoaderData.hpp"
+#include "code/aotCodeCache.hpp"
 #include "gc/shared/barrierSet.hpp"
 #include "gc/shared/barrierSetAssembler.hpp"
 #include "gc/shared/barrierSetNMethod.hpp"
@@ -405,10 +406,22 @@ void BarrierSetAssembler::c2i_entry_barrier(MacroAssembler* masm) {
 }
 
 void BarrierSetAssembler::check_oop(MacroAssembler* masm, Register obj, Register tmp1, Register tmp2, Label& error) {
+  assert_different_registers(obj, tmp1, tmp2);
   // Check if the oop is in the right area of memory
-  __ mov(tmp2, (intptr_t) Universe::verify_oop_mask());
-  __ andr(tmp1, obj, tmp2);
-  __ mov(tmp2, (intptr_t) Universe::verify_oop_bits());
+#if INCLUDE_CDS
+  if (AOTCodeCache::is_on_for_dump()) {
+    __ lea(tmp2, ExternalAddress(AOTRuntimeConstants::verify_oop_mask_address()));
+    __ ldr(tmp2, Address(tmp2));
+    __ andr(tmp1, obj, tmp2);
+    __ lea(tmp2, ExternalAddress(AOTRuntimeConstants::verify_oop_bits_address()));
+    __ ldr(tmp2, Address(tmp2));
+  } else
+#endif
+  {
+    __ mov(tmp2, (intptr_t) Universe::verify_oop_mask());
+    __ andr(tmp1, obj, tmp2);
+    __ mov(tmp2, (intptr_t) Universe::verify_oop_bits());
+  }
 
   // Compare tmp1 and tmp2.  We don't use a compare
   // instruction here because the flags register is live.

src/hotspot/cpu/aarch64/gc/shenandoah/shenandoahBarrierSetAssembler_aarch64.cpp

@@ -24,6 +24,7 @@
  *
  */
 
+#include "code/aotCodeCache.hpp"
 #include "gc/shenandoah/heuristics/shenandoahHeuristics.hpp"
 #include "gc/shenandoah/mode/shenandoahMode.hpp"
 #include "gc/shenandoah/shenandoahBarrierSet.hpp"
@@ -213,13 +214,16 @@ void ShenandoahBarrierSetAssembler::load_reference_barrier(MacroAssembler* masm,
 
   // Test for in-cset
   if (is_strong) {
+#if INCLUDE_CDS
     if (AOTCodeCache::is_on_for_dump()) {
       __ lea(rscratch2, ExternalAddress(AOTRuntimeConstants::cset_base_address()));
       __ ldr(rscratch2, Address(rscratch2));
       __ lea(rscratch1, ExternalAddress(AOTRuntimeConstants::grain_shift_address()));
       __ ldrw(rscratch1, Address(rscratch1));
       __ lsrv(rscratch1, r0, rscratch1);
-    } else {
+    } else
+#endif
+    {
       __ mov(rscratch2, ShenandoahHeap::in_cset_fast_test_addr());
       __ lsr(rscratch1, r0, ShenandoahHeapRegion::region_size_bytes_shift_jint());
     }
@@ -421,11 +425,22 @@ void ShenandoahBarrierSetAssembler::try_peek_weak_handle_in_nmethod(MacroAssembl
 }
 
 void ShenandoahBarrierSetAssembler::check_oop(MacroAssembler* masm, Register obj, Register tmp1, Register tmp2, Label& L_error) {
+  assert_different_registers(obj, tmp1, tmp2);
   // Check if the oop is in the right area of memory
-  __ mov(tmp2, (intptr_t) Universe::verify_oop_mask());
-  __ andr(tmp1, obj, tmp2);
-  __ mov(tmp2, (intptr_t) Universe::verify_oop_bits());
-
+#if INCLUDE_CDS
+  if (AOTCodeCache::is_on_for_dump()) {
+    __ lea(tmp2, ExternalAddress(AOTRuntimeConstants::verify_oop_mask_address()));
+    __ ldr(tmp2, Address(tmp2));
+    __ andr(tmp1, obj, tmp2);
+    __ lea(tmp2, ExternalAddress(AOTRuntimeConstants::verify_oop_bits_address()));
+    __ ldr(tmp2, Address(tmp2));
+  } else
+#endif
+  {
+    __ mov(tmp2, (intptr_t) Universe::verify_oop_mask());
+    __ andr(tmp1, obj, tmp2);
+    __ mov(tmp2, (intptr_t) Universe::verify_oop_bits());
+  }
   // Compare tmp1 and tmp2.  We don't use a compare
   // instruction here because the flags register is live.
   __ eor(tmp1, tmp1, tmp2);

src/hotspot/cpu/aarch64/gc/z/zBarrierSetAssembler_aarch64.cpp

@@ -22,6 +22,7 @@
  */
 
 #include "asm/macroAssembler.inline.hpp"
+#include "code/aotCodeCache.hpp"
 #include "code/codeBlob.hpp"
 #include "code/vmreg.inline.hpp"
 #include "gc/z/zAddress.hpp"
@@ -1351,6 +1352,7 @@ void ZBarrierSetAssembler::try_peek_weak_handle_in_nmethod(MacroAssembler* masm,
 }
 
 void ZBarrierSetAssembler::check_oop(MacroAssembler* masm, Register obj, Register tmp1, Register tmp2, Label& error) {
+  assert_different_registers(obj, tmp1, tmp2);
   // C1 calls verfy_oop in the middle of barriers, before they have been uncolored
   // and after being colored. Therefore, we must deal with colored oops as well.
   Label done;
@@ -1390,9 +1392,20 @@ void ZBarrierSetAssembler::check_oop(MacroAssembler* masm, Register obj, Registe
 
   __ bind(check_zaddress);
   // Check if the oop is in the right area of memory
-  __ mov(tmp1, (intptr_t) Universe::verify_oop_mask());
-  __ andr(tmp1, tmp1, obj);
-  __ mov(obj, (intptr_t) Universe::verify_oop_bits());
+#if INCLUDE_CDS
+  if (AOTCodeCache::is_on_for_dump()) {
+    __ lea(tmp1, ExternalAddress(AOTRuntimeConstants::verify_oop_mask_address()));
+    __ ldr(tmp1, Address(tmp1));
+    __ andr(tmp1, tmp1, obj);
+    __ lea(obj, ExternalAddress(AOTRuntimeConstants::verify_oop_bits_address()));
+    __ ldr(obj, Address(obj));
+  } else
+#endif
+  {
+    __ mov(tmp1, (intptr_t) Universe::verify_oop_mask());
+    __ andr(tmp1, tmp1, obj);
+    __ mov(obj, (intptr_t) Universe::verify_oop_bits());
+  }
   __ cmp(tmp1, obj);
   __ br(Assembler::NE, error);
 

src/hotspot/cpu/aarch64/gc/z/z_aarch64.ad

@@ -33,14 +33,14 @@ source %{
 
 #include "gc/z/zBarrierSetAssembler.hpp"
 
-static void z_color(MacroAssembler* masm, const MachNode* node, Register dst, Register src) {
+static void z_color(MacroAssembler* masm, Register dst, Register src) {
   assert_different_registers(src, dst);
   __ relocate(barrier_Relocation::spec(), ZBarrierRelocationFormatStoreGoodBeforeMov);
   __ movzw(dst, barrier_Relocation::unpatched);
   __ orr(dst, dst, src, Assembler::LSL, ZPointerLoadShift);
 }
 
-static void z_uncolor(MacroAssembler* masm, const MachNode* node, Register ref) {
+static void z_uncolor(MacroAssembler* masm, Register ref) {
   __ lsr(ref, ref, ZPointerLoadShift);
 }
 
@@ -50,7 +50,7 @@ static void z_keep_alive_load_barrier(MacroAssembler* masm, const MachNode* node
   __ tst(ref, tmp);
   ZLoadBarrierStubC2Aarch64* const stub = ZLoadBarrierStubC2Aarch64::create(node, ref_addr, ref);
   __ br(Assembler::NE, *stub->entry());
-  z_uncolor(masm, node, ref);
+  z_uncolor(masm, ref);
   __ bind(*stub->continuation());
 }
 
@@ -66,7 +66,7 @@ static void z_load_barrier(MacroAssembler* masm, const MachNode* node, Address r
   }
 
   if (node->barrier_data() == ZBarrierElided) {
-    z_uncolor(masm, node, ref);
+    z_uncolor(masm, ref);
     return;
   }
 
@@ -81,14 +81,14 @@ static void z_load_barrier(MacroAssembler* masm, const MachNode* node, Address r
     __ b(*stub->entry());
     __ bind(good);
   }
-  z_uncolor(masm, node, ref);
+  z_uncolor(masm, ref);
   __ bind(*stub->continuation());
 }
 
 static void z_store_barrier(MacroAssembler* masm, const MachNode* node, Address ref_addr, Register rnew_zaddress, Register rnew_zpointer, Register tmp, bool is_atomic) {
   Assembler::InlineSkippedInstructionsCounter skipped_counter(masm);
   if (node->barrier_data() == ZBarrierElided) {
-    z_color(masm, node, rnew_zpointer, rnew_zaddress);
+    z_color(masm, rnew_zpointer, rnew_zaddress);
   } else {
     bool is_native = (node->barrier_data() & ZBarrierNative) != 0;
     bool is_nokeepalive = (node->barrier_data() & ZBarrierNoKeepalive) != 0;
@@ -206,7 +206,7 @@ instruct zCompareAndSwapP(iRegINoSp res, indirect mem, iRegP oldval, iRegP newva
     guarantee($mem$$index == -1 && $mem$$disp == 0, "impossible encoding");
     Address ref_addr($mem$$Register);
     z_store_barrier(masm, this, ref_addr, $newval$$Register, $newval_tmp$$Register, rscratch2, true /* is_atomic */);
-    z_color(masm, this, $oldval_tmp$$Register, $oldval$$Register);
+    z_color(masm, $oldval_tmp$$Register, $oldval$$Register);
     __ cmpxchg($mem$$Register, $oldval_tmp$$Register, $newval_tmp$$Register, Assembler::xword, memory_order_release);
     __ cset($res$$Register, Assembler::EQ);
   %}
@@ -229,7 +229,7 @@ instruct zCompareAndSwapPAcq(iRegINoSp res, indirect mem, iRegP oldval, iRegP ne
     guarantee($mem$$index == -1 && $mem$$disp == 0, "impossible encoding");
     Address ref_addr($mem$$Register);
     z_store_barrier(masm, this, ref_addr, $newval$$Register, $newval_tmp$$Register, rscratch2, true /* is_atomic */);
-    z_color(masm, this, $oldval_tmp$$Register, $oldval$$Register);
+    z_color(masm, $oldval_tmp$$Register, $oldval$$Register);
     __ cmpxchg($mem$$Register, $oldval_tmp$$Register, $newval_tmp$$Register, Assembler::xword, memory_order_seq_cst);
     __ cset($res$$Register, Assembler::EQ);
   %}
@@ -251,10 +251,10 @@ instruct zCompareAndExchangeP(iRegPNoSp res, indirect mem, iRegP oldval, iRegP n
     guarantee($mem$$index == -1 && $mem$$disp == 0, "impossible encoding");
     Address ref_addr($mem$$Register);
     z_store_barrier(masm, this, ref_addr, $newval$$Register, $newval_tmp$$Register, rscratch2, true /* is_atomic */);
-    z_color(masm, this, $oldval_tmp$$Register, $oldval$$Register);
+    z_color(masm, $oldval_tmp$$Register, $oldval$$Register);
     __ cmpxchg($mem$$Register, $oldval_tmp$$Register, $newval_tmp$$Register, Assembler::xword,
                memory_order_release, $res$$Register);
-    z_uncolor(masm, this, $res$$Register);
+    z_uncolor(masm, $res$$Register);
   %}
 
   ins_pipe(pipe_slow);
@@ -274,10 +274,10 @@ instruct zCompareAndExchangePAcq(iRegPNoSp res, indirect mem, iRegP oldval, iReg
     guarantee($mem$$index == -1 && $mem$$disp == 0, "impossible encoding");
     Address ref_addr($mem$$Register);
     z_store_barrier(masm, this, ref_addr, $newval$$Register, $newval_tmp$$Register, rscratch2, true /* is_atomic */);
-    z_color(masm, this, $oldval_tmp$$Register, $oldval$$Register);
+    z_color(masm, $oldval_tmp$$Register, $oldval$$Register);
     __ cmpxchg($mem$$Register, $oldval_tmp$$Register, $newval_tmp$$Register, Assembler::xword,
                memory_order_seq_cst, $res$$Register);
-    z_uncolor(masm, this, $res$$Register);
+    z_uncolor(masm, $res$$Register);
   %}
 
   ins_pipe(pipe_slow);
@@ -295,7 +295,7 @@ instruct zGetAndSetP(indirect mem, iRegP newv, iRegPNoSp prev, rFlagsReg cr) %{
   ins_encode %{
     z_store_barrier(masm, this, Address($mem$$Register), $newv$$Register, $prev$$Register, rscratch2, true /* is_atomic */);
     __ atomic_xchg($prev$$Register, $prev$$Register, $mem$$Register);
-    z_uncolor(masm, this, $prev$$Register);
+    z_uncolor(masm, $prev$$Register);
   %}
 
   ins_pipe(pipe_serial);
@@ -313,7 +313,7 @@ instruct zGetAndSetPAcq(indirect mem, iRegP newv, iRegPNoSp prev, rFlagsReg cr)
   ins_encode %{
     z_store_barrier(masm, this, Address($mem$$Register), $newv$$Register, $prev$$Register, rscratch2, true /* is_atomic */);
     __ atomic_xchgal($prev$$Register, $prev$$Register, $mem$$Register);
-    z_uncolor(masm, this, $prev$$Register);
+    z_uncolor(masm, $prev$$Register);
   %}
 
   ins_pipe(pipe_serial);

src/hotspot/cpu/aarch64/globals_aarch64.hpp

@@ -42,7 +42,7 @@ define_pd_global(size_t, CodeCacheSegmentSize,   64);
 define_pd_global(uint, CodeEntryAlignment,       64);
 define_pd_global(intx, OptoLoopAlignment,        16);
 
-#define DEFAULT_STACK_YELLOW_PAGES (2)
+#define DEFAULT_STACK_YELLOW_PAGES (NOT_WINDOWS(2) WINDOWS_ONLY(3))
 #define DEFAULT_STACK_RED_PAGES (1)
 // Java_java_net_SocketOutputStream_socketWrite0() uses a 64k buffer on the
 // stack if compiled for unix and LP64. To pass stack overflow tests we need

src/hotspot/cpu/aarch64/interpreterRT_aarch64.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2003, 2025, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2026, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2014, 2020, Red Hat Inc. All rights reserved.
  * Copyright (c) 2021, Azul Systems, Inc. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
@@ -181,7 +181,7 @@ void InterpreterRuntime::SignatureHandlerGenerator::generate(uint64_t fingerprin
   __ lea(r0, ExternalAddress(Interpreter::result_handler(method()->result_type())));
   __ ret(lr);
 
-  __ flush();
+  __ invalidate_icache();
 }
 
 

src/hotspot/cpu/aarch64/jniFastGetField_aarch64.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2004, 2025, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2004, 2026, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2014, 2020, Red Hat Inc. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
@@ -206,7 +206,7 @@ address JNI_FastGetField::generate_fast_get_int_field0(BasicType type) {
     __ leave();
     __ ret(lr);
   }
-  __ flush ();
+  __ invalidate_icache();
 
   return fast_entry;
 }

src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp

@@ -1972,7 +1972,9 @@ void MacroAssembler::verify_secondary_supers_table(Register r_sub_klass,
     mov(r1, r_sub_klass);           // r1 <- r4
     mov(r2, /*expected*/rscratch1); // r2 <- r8
     mov(r3, result);                // r3 <- r5
-    mov(r4, (address)("mismatch")); // r4 <- const
+    const char* msg = "mismatch";
+    const char* str = (code_section()->scratch_emit()) ? msg : AOTCodeCache::add_C_string(msg);
+    lea(r4, ExternalAddress((address)str)); // r4 <- const
     rt_call(CAST_FROM_FN_PTR(address, Klass::on_secondary_supers_verification_failure), rscratch2);
     should_not_reach_here();
   }
@@ -2025,7 +2027,15 @@ void MacroAssembler::_verify_oop(Register reg, const char* s, const char* file,
     ResourceMark rm;
     stringStream ss;
     ss.print("verify_oop: %s: %s (%s:%d)", reg->name(), s, file, line);
-    b = code_string(ss.as_string());
+#if INCLUDE_CDS
+    if (AOTCodeCache::is_on_for_dump() && !code_section()->scratch_emit()) {
+      // this will duplicate string to preserve it
+      b = AOTCodeCache::add_C_string(ss.as_string());
+    } else
+#endif
+    {
+      b = code_string(ss.as_string());
+    }
   }
   BLOCK_COMMENT("verify_oop {");
 
@@ -2035,7 +2045,7 @@ void MacroAssembler::_verify_oop(Register reg, const char* s, const char* file,
   stp(rscratch2, lr, Address(pre(sp, -2 * wordSize)));
 
   mov(r0, reg);
-  movptr(rscratch1, (uintptr_t)(address)b);
+  lea(rscratch1, ExternalAddress((address)b));
 
   // call indirectly to solve generation ordering problem
   lea(rscratch2, RuntimeAddress(StubRoutines::verify_oop_subroutine_entry_address()));
@@ -2061,7 +2071,15 @@ void MacroAssembler::_verify_oop_addr(Address addr, const char* s, const char* f
     ResourceMark rm;
     stringStream ss;
     ss.print("verify_oop_addr: %s (%s:%d)", s, file, line);
-    b = code_string(ss.as_string());
+#if INCLUDE_CDS
+    if (AOTCodeCache::is_on_for_dump() && !code_section()->scratch_emit()) {
+      // this will duplicate string to preserve it
+      b = AOTCodeCache::add_C_string(ss.as_string());
+    } else
+#endif
+    {
+      b = code_string(ss.as_string());
+    }
   }
   BLOCK_COMMENT("verify_oop_addr {");
 
@@ -2078,7 +2096,7 @@ void MacroAssembler::_verify_oop_addr(Address addr, const char* s, const char* f
   } else {
     ldr(r0, addr);
   }
-  movptr(rscratch1, (uintptr_t)(address)b);
+  lea(rscratch1, ExternalAddress((address)b));
 
   // call indirectly to solve generation ordering problem
   lea(rscratch2, RuntimeAddress(StubRoutines::verify_oop_subroutine_entry_address()));
@@ -2396,16 +2414,6 @@ void MacroAssembler::test_field_is_flat(Register flags, Register temp_reg, Label
 void MacroAssembler::test_oop_prototype_bit(Register oop, Register temp_reg, int32_t test_bit, bool jmp_set, Label& jmp_label) {
   // load mark word
   ldr(temp_reg, Address(oop, oopDesc::mark_offset_in_bytes()));
-  if (!UseObjectMonitorTable) {
-    Label test_mark_word;
-    // check displaced
-    tst(temp_reg, markWord::unlocked_value);
-    br(Assembler::NE, test_mark_word);
-    // slow path use klass prototype
-    load_prototype_header(temp_reg, oop);
-
-    bind(test_mark_word);
-  }
   andr(temp_reg, temp_reg, test_bit);
   if (jmp_set) {
     cbnz(temp_reg, jmp_label);
@@ -7885,10 +7893,8 @@ void MacroAssembler::fast_lock(Register basic_lock, Register obj, Register t1, R
   // instruction emitted as it is part of C1's null check semantics.
   ldr(mark, Address(obj, oopDesc::mark_offset_in_bytes()));
 
-  if (UseObjectMonitorTable) {
-    // Clear cache in case fast locking succeeds or we need to take the slow-path.
-    str(zr, Address(basic_lock, BasicObjectLock::lock_offset() + in_ByteSize((BasicLock::object_monitor_cache_offset_in_bytes()))));
-  }
+  // Clear cache in case fast locking succeeds or we need to take the slow-path.
+  str(zr, Address(basic_lock, BasicObjectLock::lock_offset() + in_ByteSize((BasicLock::object_monitor_cache_offset_in_bytes()))));
 
   if (DiagnoseSyncOnValueBasedClasses != 0) {
     load_klass(t1, obj, rscratch1);

src/hotspot/cpu/aarch64/runtime_aarch64.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2003, 2025, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2026, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2014, Red Hat Inc. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
@@ -249,8 +249,7 @@ UncommonTrapBlob* OptoRuntime::generate_uncommon_trap_blob() {
   // Jump to interpreter
   __ ret(lr);
 
-  // Make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   UncommonTrapBlob *ut_blob = UncommonTrapBlob::create(&buffer, oop_maps,
                                                        SimpleRuntimeFrame::framesize >> 1);
@@ -391,8 +390,7 @@ ExceptionBlob* OptoRuntime::generate_exception_blob() {
 
   __ br(r8);
 
-  // Make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   // Set exception blob
   ExceptionBlob* ex_blob = ExceptionBlob::create(&buffer, oop_maps, SimpleRuntimeFrame::framesize >> 1);
@@ -400,5 +398,3 @@ ExceptionBlob* OptoRuntime::generate_exception_blob() {
   return ex_blob;
 }
 #endif // COMPILER2
-
-

src/hotspot/cpu/aarch64/sharedRuntime_aarch64.cpp

@@ -1612,7 +1612,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
     assert(vep_offset != -1,        "Must be set");
 #endif
 
-    __ flush();
+    // Code will be copied. No ICache sync required.
     nmethod* nm = nmethod::new_native_nmethod(method,
                                               compile_id,
                                               masm->code(),
@@ -1646,7 +1646,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
                          in_sig_bt,
                          in_regs);
     int frame_complete = ((intptr_t)__ pc()) - start;  // not complete, period
-    __ flush();
+    // Code will be copied. No ICache sync required.
     int stack_slots = SharedRuntime::out_preserve_stack_slots();  // no out slots at all, actually
     return nmethod::new_native_nmethod(method,
                                        compile_id,
@@ -2047,14 +2047,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
 
   Label safepoint_in_progress, safepoint_in_progress_done;
 
-  // Switch thread to "native transition" state before reading the synchronization state.
-  // This additional state is necessary because reading and testing the synchronization
-  // state is not atomic w.r.t. GC, as this scenario demonstrates:
-  //     Java thread A, in _thread_in_native state, loads _not_synchronized and is preempted.
-  //     VM thread changes sync state to synchronizing and suspends threads for GC.
-  //     Thread A is resumed to finish this native method, but doesn't block here since it
-  //     didn't see any synchronization is progress, and escapes.
-  __ mov(rscratch1, _thread_in_native_trans);
+  __ mov(rscratch1, _thread_in_vm);
 
   __ strw(rscratch1, Address(rthread, JavaThread::thread_state_offset()));
 
@@ -2323,7 +2316,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
     }
   }
 
-  __ flush();
+  // Code will be copied. No ICache sync required.
 
   nmethod *nm = nmethod::new_native_nmethod(method,
                                             compile_id,
@@ -2664,8 +2657,7 @@ void SharedRuntime::generate_deopt_blob() {
   // Jump to interpreter
   __ ret(lr);
 
-  // Make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   _deopt_blob = DeoptimizationBlob::create(&buffer, oop_maps, 0, exception_offset, reexecute_offset, frame_size_in_words);
   _deopt_blob->set_unpack_with_exception_in_tls_offset(exception_in_tls_offset);
@@ -2813,8 +2805,7 @@ SafepointBlob* SharedRuntime::generate_handler_blob(StubId id, address call_ptr)
   __ stop("Attempting to adjust pc to skip safepoint poll but the return point is not what we expected");
 #endif
 
-  // Make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   // Fill-out other meta info
   SafepointBlob* sp_blob = SafepointBlob::create(&buffer, oop_maps, frame_size_in_words);
@@ -2909,9 +2900,7 @@ RuntimeStub* SharedRuntime::generate_resolve_blob(StubId id, address destination
   __ ldr(r0, Address(rthread, Thread::pending_exception_offset()));
   __ far_jump(RuntimeAddress(StubRoutines::forward_exception_entry()));
 
-  // -------------
-  // make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   // return the  blob
   // frame_size_words or bytes??
@@ -3065,7 +3054,7 @@ BufferedInlineTypeBlob* SharedRuntime::generate_buffered_inline_type_adapter(con
 
   __ ret(lr);
 
-  __ flush();
+  // Code will be copied. No ICache sync required.
 
   return BufferedInlineTypeBlob::create(&buffer, pack_fields_off, pack_fields_jobject_off, unpack_fields_off);
 }
@@ -3316,9 +3305,7 @@ RuntimeStub* SharedRuntime::generate_return_value_stub(address destination) {
   __ leave();
   __ far_jump(RuntimeAddress(StubRoutines::forward_exception_entry()));
 
-  // -------------
-  // make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   RuntimeStub* stub = RuntimeStub::new_runtime_stub(name, &code, frame_complete, frame_size_in_words, oop_maps, false);
   AOTCodeCache::store_code_blob(*stub, AOTCodeEntry::SharedBlob, StubInfo::blob(id));

src/hotspot/cpu/aarch64/stubGenerator_aarch64.cpp

@@ -827,7 +827,7 @@ class StubGenerator: public StubCodeGenerator {
     assert(frame::arg_reg_save_area_bytes == 0, "not expecting frame reg save area");
 #endif
     BLOCK_COMMENT("call MacroAssembler::debug");
-    __ mov(rscratch1, CAST_FROM_FN_PTR(address, MacroAssembler::debug64));
+    __ lea(rscratch1, RuntimeAddress(CAST_FROM_FN_PTR(address, MacroAssembler::debug64)));
     __ blr(rscratch1);
     __ hlt(0);
 
@@ -12826,7 +12826,7 @@ class StubGenerator: public StubCodeGenerator {
     // Native caller has no idea how to handle exceptions,
     // so we just crash here. Up to callee to catch exceptions.
     __ verify_oop(r0);
-    __ movptr(rscratch1, CAST_FROM_FN_PTR(uint64_t, UpcallLinker::handle_uncaught_exception));
+    __ lea(rscratch1, RuntimeAddress(CAST_FROM_FN_PTR(address, UpcallLinker::handle_uncaught_exception)));
     __ blr(rscratch1);
     __ should_not_reach_here();
 

src/hotspot/cpu/aarch64/templateInterpreterGenerator_aarch64.cpp

@@ -1422,7 +1422,7 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) {
   __ verify_sve_vector_length();
 
   // change thread state
-  __ mov(rscratch1, _thread_in_native_trans);
+  __ mov(rscratch1, _thread_in_vm);
   __ lea(rscratch2, Address(rthread, JavaThread::thread_state_offset()));
   __ stlrw(rscratch1, rscratch2);
 

src/hotspot/cpu/aarch64/upcallLinker_aarch64.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2020, 2025, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2026, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2019, 2022, Arm Limited. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
@@ -310,7 +310,7 @@ address UpcallLinker::make_upcall_stub(jobject receiver, Symbol* signature,
 
   //////////////////////////////////////////////////////////////////////////////
 
-  _masm->flush();
+  // Code will be copied. No ICache sync required.
 
 #ifndef PRODUCT
   stringStream ss;

src/hotspot/cpu/aarch64/vtableStubs_aarch64.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2003, 2025, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2026, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2014, Red Hat Inc. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
@@ -132,7 +132,7 @@ VtableStub* VtableStubs::create_vtable_stub(int vtable_index, bool caller_is_c1)
   __ ldr(rscratch1, Address(rmethod, entry_offset));
   __ br(rscratch1);
 
-  masm->flush();
+  masm->invalidate_icache();
   bookkeeping(masm, tty, s, npe_addr, ame_addr, true, vtable_index, slop_bytes, 0);
 
   return s;
@@ -233,7 +233,7 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index, bool caller_is_c1)
   assert(SharedRuntime::get_handle_wrong_method_stub() != nullptr, "check initialization order");
   __ far_jump(RuntimeAddress(SharedRuntime::get_handle_wrong_method_stub()));
 
-  masm->flush();
+  masm->invalidate_icache();
   bookkeeping(masm, tty, s, npe_addr, ame_addr, false, itable_index, slop_bytes, 0);
 
   return s;

src/hotspot/cpu/arm/bytes_arm.hpp

@@ -1,180 +0,0 @@
-/*
- * Copyright (c) 2008, 2022, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#ifndef CPU_ARM_BYTES_ARM_HPP
-#define CPU_ARM_BYTES_ARM_HPP
-
-#include "memory/allStatic.hpp"
-#include "utilities/macros.hpp"
-
-#ifndef VM_LITTLE_ENDIAN
-#define VM_LITTLE_ENDIAN  1
-#endif
-
-class Bytes: AllStatic {
-
- public:
-  static inline u2 get_Java_u2(address p) {
-    return (u2(p[0]) << 8) | u2(p[1]);
-  }
-
-  static inline u4 get_Java_u4(address p) {
-    return u4(p[0]) << 24 |
-           u4(p[1]) << 16 |
-           u4(p[2]) <<  8 |
-           u4(p[3]);
-  }
-
-  static inline u8 get_Java_u8(address p) {
-    return u8(p[0]) << 56 |
-           u8(p[1]) << 48 |
-           u8(p[2]) << 40 |
-           u8(p[3]) << 32 |
-           u8(p[4]) << 24 |
-           u8(p[5]) << 16 |
-           u8(p[6]) <<  8 |
-           u8(p[7]);
-  }
-
-  static inline void put_Java_u2(address p, u2 x) {
-    p[0] = x >> 8;
-    p[1] = x;
-  }
-
-  static inline void put_Java_u4(address p, u4 x) {
-    ((u1*)p)[0] = x >> 24;
-    ((u1*)p)[1] = x >> 16;
-    ((u1*)p)[2] = x >>  8;
-    ((u1*)p)[3] = x;
-  }
-
-  static inline void put_Java_u8(address p, u8 x) {
-    ((u1*)p)[0] = x >> 56;
-    ((u1*)p)[1] = x >> 48;
-    ((u1*)p)[2] = x >> 40;
-    ((u1*)p)[3] = x >> 32;
-    ((u1*)p)[4] = x >> 24;
-    ((u1*)p)[5] = x >> 16;
-    ((u1*)p)[6] = x >>  8;
-    ((u1*)p)[7] = x;
-  }
-
-#ifdef VM_LITTLE_ENDIAN
-
-  static inline u2 get_native_u2(address p) {
-    return (intptr_t(p) & 1) == 0 ? *(u2*)p : u2(p[0]) | (u2(p[1]) << 8);
-  }
-
-  static inline u4 get_native_u4(address p) {
-    switch (intptr_t(p) & 3) {
-      case 0:  return *(u4*)p;
-      case 2:  return u4(((u2*)p)[0]) |
-                      u4(((u2*)p)[1]) << 16;
-      default: return u4(p[0])       |
-                      u4(p[1]) <<  8 |
-                      u4(p[2]) << 16 |
-                      u4(p[3]) << 24;
-    }
-  }
-
-  static inline u8 get_native_u8(address p) {
-    switch (intptr_t(p) & 7) {
-      case 0:  return *(u8*)p;
-      case 4:  return u8(((u4*)p)[0]) |
-                      u8(((u4*)p)[1]) << 32;
-      case 2:  return u8(((u2*)p)[0])       |
-                      u8(((u2*)p)[1]) << 16 |
-                      u8(((u2*)p)[2]) << 32 |
-                      u8(((u2*)p)[3]) << 48;
-      default: return u8(p[0])       |
-                      u8(p[1]) <<  8 |
-                      u8(p[2]) << 16 |
-                      u8(p[3]) << 24 |
-                      u8(p[4]) << 32 |
-                      u8(p[5]) << 40 |
-                      u8(p[6]) << 48 |
-                      u8(p[7]) << 56;
-    }
-  }
-
-  static inline void put_native_u2(address p, u2 x) {
-    if ((intptr_t(p) & 1) == 0) {
-      *(u2*)p = x;
-    } else {
-      p[0] = x;
-      p[1] = x >> 8;
-    }
-  }
-
-  static inline void put_native_u4(address p, u4 x) {
-    switch (intptr_t(p) & 3) {
-      case 0:  *(u4*)p = x;
-               break;
-      case 2:  ((u2*)p)[0] = x;
-               ((u2*)p)[1] = x >> 16;
-               break;
-      default: ((u1*)p)[0] = x;
-               ((u1*)p)[1] = x >>  8;
-               ((u1*)p)[2] = x >> 16;
-               ((u1*)p)[3] = x >> 24;
-               break;
-    }
-  }
-
-  static inline void put_native_u8(address p, u8 x) {
-    switch (intptr_t(p) & 7) {
-      case 0:  *(u8*)p = x;
-               break;
-      case 4:  ((u4*)p)[0] = x;
-               ((u4*)p)[1] = x >> 32;
-               break;
-      case 2:  ((u2*)p)[0] = x;
-               ((u2*)p)[1] = x >> 16;
-               ((u2*)p)[2] = x >> 32;
-               ((u2*)p)[3] = x >> 48;
-               break;
-      default: ((u1*)p)[0] = x;
-               ((u1*)p)[1] = x >>  8;
-               ((u1*)p)[2] = x >> 16;
-               ((u1*)p)[3] = x >> 24;
-               ((u1*)p)[4] = x >> 32;
-               ((u1*)p)[5] = x >> 40;
-               ((u1*)p)[6] = x >> 48;
-               ((u1*)p)[7] = x >> 56;
-    }
-  }
-
-#else
-
-  static inline u2 get_native_u2(address p) { return get_Java_u2(p); }
-  static inline u4 get_native_u4(address p) { return get_Java_u4(p); }
-  static inline u8 get_native_u8(address p) { return get_Java_u8(p); }
-  static inline void put_native_u2(address p, u2 x) { put_Java_u2(p, x); }
-  static inline void put_native_u4(address p, u4 x) { put_Java_u4(p, x); }
-  static inline void put_native_u8(address p, u8 x) { put_Java_u8(p, x); }
-
-#endif // VM_LITTLE_ENDIAN
-};
-
-#endif // CPU_ARM_BYTES_ARM_HPP

src/hotspot/cpu/arm/jniFastGetField_arm.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2008, 2025, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2008, 2026, Oracle and/or its affiliates. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * This code is free software; you can redistribute it and/or modify it
@@ -210,7 +210,7 @@ address JNI_FastGetField::generate_fast_get_int_field0(BasicType type) {
 
   __ bind_literal(safepoint_counter_addr);
 
-  __ flush();
+  __ invalidate_icache();
 
   guarantee((__ pc() - fast_entry) <= BUFFER_SIZE, "BUFFER_SIZE too small");
 

src/hotspot/cpu/arm/macroAssembler_arm.hpp

@@ -449,7 +449,7 @@ public:
   int should_not_call_this() {
     raw_push(FP, LR);
     should_not_reach_here();
-    flush();
+    invalidate_icache();
     return 2; // frame_size_in_words (FP+LR)
   }
 

src/hotspot/cpu/arm/runtime_arm.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2008, 2025, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2008, 2026, Oracle and/or its affiliates. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * This code is free software; you can redistribute it and/or modify it
@@ -176,7 +176,7 @@ UncommonTrapBlob* OptoRuntime::generate_uncommon_trap_blob() {
   __ mov(SP, FP);
   __ pop(RegisterSet(FP) | RegisterSet(PC));
 
-  masm->flush();
+  masm->invalidate_icache();
   return UncommonTrapBlob::create(&buffer, nullptr, 2 /* LR+FP */);
 }
 
@@ -280,7 +280,7 @@ ExceptionBlob* OptoRuntime::generate_exception_blob() {
 
   // -------------
   // make sure all code is generated
-  masm->flush();
+  masm->invalidate_icache();
 
   return ExceptionBlob::create(&buffer, oop_maps, framesize_in_words);
 }

src/hotspot/cpu/arm/sharedRuntime_arm.cpp

@@ -850,7 +850,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
                          in_sig_bt,
                          in_regs);
     int frame_complete = ((intptr_t)__ pc()) - start;  // not complete, period
-    __ flush();
+    __ invalidate_icache();
     int stack_slots = SharedRuntime::out_preserve_stack_slots();  // no out slots at all, actually
     return nmethod::new_native_nmethod(method,
                                        compile_id,
@@ -1263,9 +1263,9 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
     __ c2bool(R0);
   }
 
-  // Do a safepoint check while thread is in transition state
+  // Do a safepoint check
   Label call_safepoint_runtime, return_to_java;
-  __ mov(Rtemp, _thread_in_native_trans);
+  __ mov(Rtemp, _thread_in_vm);
   __ str_32(Rtemp, Address(Rthread, JavaThread::thread_state_offset()));
 
   // make sure the store is observed before reading the SafepointSynchronize state and further mem refs
@@ -1292,6 +1292,9 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
 
   Label slow_unlock, unlock_done;
   if (method->is_synchronized()) {
+    // Get locked oop from the handle we passed to jni
+    __ ldr(sync_obj, Address(sync_handle));
+
     log_trace(fastlock)("SharedRuntime unlock fast");
     __ fast_unlock(sync_obj, R2 /* t1 */, tmp /* t2 */, Rtemp /* t3 */,
                    7 /* savemask */, slow_unlock);
@@ -1382,7 +1385,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
     __ b(unlock_done);
   }
 
-  __ flush();
+  __ invalidate_icache();
   return nmethod::new_native_nmethod(method,
                                      compile_id,
                                      masm->code(),
@@ -1651,7 +1654,7 @@ void SharedRuntime::generate_deopt_blob() {
 
   __ pop(RegisterSet(FP) | RegisterSet(PC));
 
-  __ flush();
+  __ invalidate_icache();
 
   _deopt_blob = DeoptimizationBlob::create(&buffer, oop_maps, 0, exception_offset,
                                            reexecute_offset, frame_size_in_words);
@@ -1731,7 +1734,7 @@ SafepointBlob* SharedRuntime::generate_handler_blob(StubId id, address call_ptr)
 
   __ jump(StubRoutines::forward_exception_entry(), relocInfo::runtime_call_type, Rtemp);
 
-  __ flush();
+  __ invalidate_icache();
 
   return SafepointBlob::create(&buffer, oop_maps, frame_size_words);
 }
@@ -1791,7 +1794,7 @@ RuntimeStub* SharedRuntime::generate_resolve_blob(StubId id, address destination
   __ mov(Rexception_pc, LR);
   __ jump(StubRoutines::forward_exception_entry(), relocInfo::runtime_call_type, Rtemp);
 
-  __ flush();
+  __ invalidate_icache();
 
   return RuntimeStub::new_runtime_stub(name, &buffer, frame_complete, frame_size_words, oop_maps, true);
 }

src/hotspot/cpu/arm/templateInterpreterGenerator_arm.cpp

@@ -1014,7 +1014,7 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) {
   }
 
   // Do safepoint check
-  __ mov(Rtemp, _thread_in_native_trans);
+  __ mov(Rtemp, _thread_in_vm);
   __ str_32(Rtemp, Address(Rthread, JavaThread::thread_state_offset()));
 
   // Force this write out before the read below

src/hotspot/cpu/arm/vtableStubs_arm.cpp

@@ -110,7 +110,7 @@ VtableStub* VtableStubs::create_vtable_stub(int vtable_index, bool caller_is_c1)
   address ame_addr = __ pc();
   __ ldr(PC, Address(Rmethod, Method::from_compiled_offset()));
 
-  masm->flush();
+  masm->invalidate_icache();
   bookkeeping(masm, tty, s, npe_addr, ame_addr, true, vtable_index, slop_bytes, 0);
 
   return s;
@@ -205,7 +205,7 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index, bool caller_is_c1)
   assert(SharedRuntime::get_handle_wrong_method_stub() != nullptr, "check initialization order");
   __ jump(SharedRuntime::get_handle_wrong_method_stub(), relocInfo::runtime_call_type, Rtemp);
 
-  masm->flush();
+  masm->invalidate_icache();
   bookkeeping(masm, tty, s, npe_addr, ame_addr, false, itable_index, slop_bytes, 0);
 
   return s;

src/hotspot/cpu/ppc/assembler_ppc.hpp

@@ -539,6 +539,10 @@ class Assembler : public AbstractAssembler {
     STXVL_OPCODE   = (31u << OPCODE_SHIFT |  397u << 1),
     LXVD2X_OPCODE  = (31u << OPCODE_SHIFT |  844u << 1),
     STXVD2X_OPCODE = (31u << OPCODE_SHIFT |  972u << 1),
+    LXVW4X_OPCODE  = (31u << OPCODE_SHIFT |  780u << 1),
+    STXVW4X_OPCODE = (31u << OPCODE_SHIFT |  908u << 1),
+    LXVB16X_OPCODE = (31u << OPCODE_SHIFT |  876u << 1),
+    STXVB16X_OPCODE= (31u << OPCODE_SHIFT | 1004u << 1),
     MTVSRD_OPCODE  = (31u << OPCODE_SHIFT |  179u << 1),
     MTVSRDD_OPCODE = (31u << OPCODE_SHIFT |  435u << 1),
     MTVSRWZ_OPCODE = (31u << OPCODE_SHIFT |  243u << 1),
@@ -1365,10 +1369,6 @@ class Assembler : public AbstractAssembler {
     return (0 == addr % a);
   }
 
-  void flush() {
-    AbstractAssembler::flush();
-  }
-
   inline void emit_int32(int);  // shadows AbstractAssembler::emit_int32
   inline void emit_data(int);
   inline void emit_data(int, RelocationHolder const&);
@@ -2386,8 +2386,17 @@ class Assembler : public AbstractAssembler {
   inline void lxvd2x(   VectorSRegister d, Register a, Register b);
   inline void stxvd2x(  VectorSRegister d, Register a);
   inline void stxvd2x(  VectorSRegister d, Register a, Register b);
+  inline void lxvw4x(   VectorSRegister d, Register a);
+  inline void lxvw4x(   VectorSRegister d, Register a, Register b);
+  inline void stxvw4x(  VectorSRegister d, Register a);
+  inline void stxvw4x(  VectorSRegister d, Register a, Register b);
 
   // Power9
+  inline void lxvb16x(  VectorSRegister d, Register a);
+  inline void lxvb16x(  VectorSRegister d, Register a, Register b);
+  inline void stxvb16x( VectorSRegister d, Register a);
+  inline void stxvb16x( VectorSRegister d, Register a, Register b);
+
   inline void lxv(      VectorSRegister d, int si16, Register a);
   inline void stxv(     VectorSRegister d, int si16, Register a);
   inline void lxvx(     VectorSRegister d, Register a, Register b);
@@ -2590,6 +2599,15 @@ class Assembler : public AbstractAssembler {
   inline void vec_perm(VectorRegister first_dest, VectorRegister second, VectorRegister perm);
   inline void vec_perm(VectorRegister dest, VectorRegister first, VectorRegister second, VectorRegister perm);
 
+  // Load/Store unaligned vectors with offs (multiple of 16). Byte versions require vp for Power8 LE.
+  inline void load_byte_vector_unaligned(VectorRegister dest, int offs, Register base, Register tmp,
+                                         VectorRegister vp); // vp should be pre-computed (see generator below)
+  inline void store_byte_vector_unaligned(VectorRegister val, int offs, Register base, Register tmp,
+                                          VectorRegister vp, VectorRegister vtmp = vnoreg); // clobbers val if no vtmp provided
+  inline void compute_vp_for_byte_vector_unaligned(VectorRegister dest, VectorRegister vtmp);
+  inline void load_word_vector_unaligned(VectorRegister dest, int offs, Register base, Register tmp);
+  inline void store_word_vector_unaligned(VectorRegister val, int offs, Register base, Register tmp);
+
   // RegisterOrConstant versions.
   // These emitters choose between the versions using two registers and
   // those with register and immediate, depending on the content of roc.

src/hotspot/cpu/ppc/assembler_ppc.inline.hpp

@@ -856,6 +856,14 @@ inline void Assembler::lxvd2x(  VectorSRegister d, Register s1)              { e
 inline void Assembler::lxvd2x(  VectorSRegister d, Register s1, Register s2) { emit_int32( LXVD2X_OPCODE  | vsrt(d) | ra0mem(s1) | rb(s2)); }
 inline void Assembler::stxvd2x( VectorSRegister d, Register s1)              { emit_int32( STXVD2X_OPCODE | vsrs(d) | ra(0) | rb(s1)); }
 inline void Assembler::stxvd2x( VectorSRegister d, Register s1, Register s2) { emit_int32( STXVD2X_OPCODE | vsrs(d) | ra0mem(s1) | rb(s2)); }
+inline void Assembler::lxvw4x(  VectorSRegister d, Register s1)              { emit_int32( LXVW4X_OPCODE  | vsrt(d) | ra(0) | rb(s1)); }
+inline void Assembler::lxvw4x(  VectorSRegister d, Register s1, Register s2) { emit_int32( LXVW4X_OPCODE  | vsrt(d) | ra0mem(s1) | rb(s2)); }
+inline void Assembler::stxvw4x( VectorSRegister d, Register s1)              { emit_int32( STXVW4X_OPCODE | vsrs(d) | ra(0) | rb(s1)); }
+inline void Assembler::stxvw4x( VectorSRegister d, Register s1, Register s2) { emit_int32( STXVW4X_OPCODE | vsrs(d) | ra0mem(s1) | rb(s2)); }
+inline void Assembler::lxvb16x( VectorSRegister d, Register s1)              { emit_int32( LXVB16X_OPCODE | vsrt(d) | ra(0) | rb(s1)); }
+inline void Assembler::lxvb16x( VectorSRegister d, Register s1, Register s2) { emit_int32( LXVB16X_OPCODE | vsrt(d) | ra0mem(s1) | rb(s2)); }
+inline void Assembler::stxvb16x(VectorSRegister d, Register s1)              { emit_int32( STXVB16X_OPCODE| vsrs(d) | ra(0) | rb(s1)); }
+inline void Assembler::stxvb16x(VectorSRegister d, Register s1, Register s2) { emit_int32( STXVB16X_OPCODE| vsrs(d) | ra0mem(s1) | rb(s2)); }
 inline void Assembler::mtvsrd(  VectorSRegister d, Register a)               { emit_int32( MTVSRD_OPCODE  | vsrt(d)  | ra(a)); }
 inline void Assembler::mtvsrdd( VectorSRegister d, Register a, Register b)   { emit_int32( MTVSRDD_OPCODE | vsrt(d)  | ra(a) | rb(b)); }
 inline void Assembler::mfvsrd(  Register d, VectorSRegister a)               { emit_int32( MFVSRD_OPCODE  | vsrs(a)  | ra(d)); }
@@ -1232,6 +1240,108 @@ inline void Assembler::vec_perm(VectorRegister dest, VectorRegister first, Vecto
 #endif
 }
 
+inline void Assembler::load_byte_vector_unaligned(VectorRegister dest, int offs, Register base, Register tmp,
+                                                  VectorRegister vp) {
+  VectorSRegister vsr = dest->to_vsr();
+  if (PowerArchitecturePPC64 >= 9) {
+#if !defined(VM_LITTLE_ENDIAN)
+    lxv(vsr, offs, base); // all vector load/store instructions use the same byte order on BE
+#else
+    if (offs == 0) {
+      lxvb16x(vsr, base);
+    } else {
+      li(tmp, offs);
+      lxvb16x(vsr, base, tmp);
+    }
+#endif
+  } else { // Power8 only supports very limited instructions
+    if (offs == 0) {
+      lxvd2x(vsr, base);
+    } else {
+      li(tmp, offs);
+      lxvd2x(vsr, base, tmp);
+    }
+#if defined(VM_LITTLE_ENDIAN)
+    // need to swap bytes in both double-words
+    vperm(dest, dest, dest, vp);
+#endif
+  }
+}
+
+inline void Assembler::store_byte_vector_unaligned(VectorRegister val, int offs, Register base, Register tmp,
+                                                   VectorRegister vp, VectorRegister vtmp) {
+  VectorSRegister vsr = val->to_vsr();
+  if (PowerArchitecturePPC64 >= 9) {
+#if !defined(VM_LITTLE_ENDIAN)
+    stxv(vsr, offs, base); // all vector load/store instructions use the same byte order on BE
+#else
+    if (offs == 0) {
+      stxvb16x(vsr, base);
+    } else {
+      li(tmp, offs);
+      stxvb16x(vsr, base, tmp);
+    }
+#endif
+  } else { // Power8 only supports very limited instructions
+#if defined(VM_LITTLE_ENDIAN)
+    // need to swap bytes in both double-words
+    if (vtmp != vnoreg) {
+      vperm(vtmp, val, val, vp);
+      vsr = vtmp->to_vsr();
+    } else {
+      vperm(val, val, val, vp); // clobbers val!
+    }
+#endif
+    if (offs == 0) {
+      stxvd2x(vsr, base);
+    } else {
+      li(tmp, offs);
+      stxvd2x(vsr, base, tmp);
+    }
+  }
+}
+
+inline void Assembler::compute_vp_for_byte_vector_unaligned(VectorRegister dest, VectorRegister vtmp) {
+#if defined(VM_LITTLE_ENDIAN)
+  if (PowerArchitecturePPC64 < 9) {
+    li(R0, 0);
+    vspltisb(vtmp, 7);      // vtmp = [7, ..., 7]
+    lvsl(dest, R0);         // dest = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
+    vxor(dest, dest, vtmp); // dest = [7, 6, 5, 4, 3, 2, 1, 0, 15, 14, 13, 12, 11, 10, 9, 8]
+  }
+#endif
+}
+
+inline void Assembler::load_word_vector_unaligned(VectorRegister dest, int offs, Register base, Register tmp) {
+  VectorSRegister vsr = dest->to_vsr();
+#if !defined(VM_LITTLE_ENDIAN)
+  if (PowerArchitecturePPC64 >= 9) {
+    lxv(vsr, offs, base); // all vector load/store instructions use the same byte order on BE
+  } else
+#endif
+  if (offs == 0) {
+    lxvw4x(vsr, base);
+  } else {
+    li(tmp, offs);
+    lxvw4x(vsr, base, tmp);
+  }
+}
+
+inline void Assembler::store_word_vector_unaligned(VectorRegister val, int offs, Register base, Register tmp) {
+  VectorSRegister vsr = val->to_vsr();
+#if !defined(VM_LITTLE_ENDIAN)
+  if (PowerArchitecturePPC64 >= 9) {
+    stxv(vsr, offs, base); // all vector load/store instructions use the same byte order on BE
+  } else
+#endif
+  if (offs == 0) {
+    stxvw4x(vsr, base);
+  } else {
+    li(tmp, offs);
+    stxvw4x(vsr, base, tmp);
+  }
+}
+
 inline void Assembler::load_const(Register d, void* x, Register tmp) {
    load_const(d, (long)x, tmp);
 }

src/hotspot/cpu/ppc/bytes_ppc.hpp

@@ -1,260 +0,0 @@
-/*
- * Copyright (c) 1997, 2022, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2012, 2022 SAP SE. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#ifndef CPU_PPC_BYTES_PPC_HPP
-#define CPU_PPC_BYTES_PPC_HPP
-
-#include "memory/allStatic.hpp"
-#include "utilities/byteswap.hpp"
-
-class Bytes: AllStatic {
- public:
-  // Efficient reading and writing of unaligned unsigned data in platform-specific byte ordering
-  // PowerPC needs to check for alignment.
-
-  // Can I count on address always being a pointer to an unsigned char? Yes.
-
-#if defined(VM_LITTLE_ENDIAN)
-
-  static inline u2   get_native_u2(address p) {
-    return (intptr_t(p) & 1) == 0
-             ?   *(u2*)p
-             :   ( u2(p[1]) << 8 )
-               | ( u2(p[0])      );
-  }
-
-  static inline u4   get_native_u4(address p) {
-    switch (intptr_t(p) & 3) {
-     case 0:  return *(u4*)p;
-
-     case 2:  return (  u4( ((u2*)p)[1] ) << 16  )
-                   | (  u4( ((u2*)p)[0] )        );
-
-    default:  return ( u4(p[3]) << 24 )
-                   | ( u4(p[2]) << 16 )
-                   | ( u4(p[1]) <<  8 )
-                   |   u4(p[0]);
-    }
-  }
-
-  static inline u8   get_native_u8(address p) {
-    switch (intptr_t(p) & 7) {
-      case 0:  return *(u8*)p;
-
-      case 4:  return (  u8( ((u4*)p)[1] ) << 32  )
-                    | (  u8( ((u4*)p)[0] )        );
-
-      case 2:  return (  u8( ((u2*)p)[3] ) << 48  )
-                    | (  u8( ((u2*)p)[2] ) << 32  )
-                    | (  u8( ((u2*)p)[1] ) << 16  )
-                    | (  u8( ((u2*)p)[0] )        );
-
-     default:  return ( u8(p[7]) << 56 )
-                    | ( u8(p[6]) << 48 )
-                    | ( u8(p[5]) << 40 )
-                    | ( u8(p[4]) << 32 )
-                    | ( u8(p[3]) << 24 )
-                    | ( u8(p[2]) << 16 )
-                    | ( u8(p[1]) <<  8 )
-                    |   u8(p[0]);
-    }
-  }
-
-
-
-  static inline void put_native_u2(address p, u2 x) {
-    if ( (intptr_t(p) & 1) == 0 )  *(u2*)p = x;
-    else {
-      p[1] = x >> 8;
-      p[0] = x;
-    }
-  }
-
-  static inline void put_native_u4(address p, u4 x) {
-    switch ( intptr_t(p) & 3 ) {
-    case 0:  *(u4*)p = x;
-              break;
-
-    case 2:  ((u2*)p)[1] = x >> 16;
-             ((u2*)p)[0] = x;
-             break;
-
-    default: ((u1*)p)[3] = x >> 24;
-             ((u1*)p)[2] = x >> 16;
-             ((u1*)p)[1] = x >>  8;
-             ((u1*)p)[0] = x;
-             break;
-    }
-  }
-
-  static inline void put_native_u8(address p, u8 x) {
-    switch ( intptr_t(p) & 7 ) {
-    case 0:  *(u8*)p = x;
-             break;
-
-    case 4:  ((u4*)p)[1] = x >> 32;
-             ((u4*)p)[0] = x;
-             break;
-
-    case 2:  ((u2*)p)[3] = x >> 48;
-             ((u2*)p)[2] = x >> 32;
-             ((u2*)p)[1] = x >> 16;
-             ((u2*)p)[0] = x;
-             break;
-
-    default: ((u1*)p)[7] = x >> 56;
-             ((u1*)p)[6] = x >> 48;
-             ((u1*)p)[5] = x >> 40;
-             ((u1*)p)[4] = x >> 32;
-             ((u1*)p)[3] = x >> 24;
-             ((u1*)p)[2] = x >> 16;
-             ((u1*)p)[1] = x >>  8;
-             ((u1*)p)[0] = x;
-    }
-  }
-
-  // Efficient reading and writing of unaligned unsigned data in Java byte ordering (i.e. big-endian ordering)
-  // (no byte-order reversal is needed since Power CPUs are big-endian oriented).
-  static inline u2   get_Java_u2(address p) { return byteswap(get_native_u2(p)); }
-  static inline u4   get_Java_u4(address p) { return byteswap(get_native_u4(p)); }
-  static inline u8   get_Java_u8(address p) { return byteswap(get_native_u8(p)); }
-
-  static inline void put_Java_u2(address p, u2 x)     { put_native_u2(p, byteswap(x)); }
-  static inline void put_Java_u4(address p, u4 x)     { put_native_u4(p, byteswap(x)); }
-  static inline void put_Java_u8(address p, u8 x)     { put_native_u8(p, byteswap(x)); }
-
-#else // !defined(VM_LITTLE_ENDIAN)
-
-  static inline u2   get_native_u2(address p) {
-    return (intptr_t(p) & 1) == 0
-             ?   *(u2*)p
-             :   ( u2(p[0]) << 8 )
-               | ( u2(p[1])      );
-  }
-
-  static inline u4   get_native_u4(address p) {
-    switch (intptr_t(p) & 3) {
-     case 0:  return *(u4*)p;
-
-     case 2:  return (  u4( ((u2*)p)[0] ) << 16  )
-                   | (  u4( ((u2*)p)[1] )        );
-
-    default:  return ( u4(p[0]) << 24 )
-                   | ( u4(p[1]) << 16 )
-                   | ( u4(p[2]) <<  8 )
-                   |   u4(p[3]);
-    }
-  }
-
-  static inline u8   get_native_u8(address p) {
-    switch (intptr_t(p) & 7) {
-      case 0:  return *(u8*)p;
-
-      case 4:  return (  u8( ((u4*)p)[0] ) << 32  )
-                    | (  u8( ((u4*)p)[1] )        );
-
-      case 2:  return (  u8( ((u2*)p)[0] ) << 48  )
-                    | (  u8( ((u2*)p)[1] ) << 32  )
-                    | (  u8( ((u2*)p)[2] ) << 16  )
-                    | (  u8( ((u2*)p)[3] )        );
-
-     default:  return ( u8(p[0]) << 56 )
-                    | ( u8(p[1]) << 48 )
-                    | ( u8(p[2]) << 40 )
-                    | ( u8(p[3]) << 32 )
-                    | ( u8(p[4]) << 24 )
-                    | ( u8(p[5]) << 16 )
-                    | ( u8(p[6]) <<  8 )
-                    |   u8(p[7]);
-    }
-  }
-
-
-
-  static inline void put_native_u2(address p, u2 x) {
-    if ( (intptr_t(p) & 1) == 0 ) { *(u2*)p = x; }
-    else {
-      p[0] = x >> 8;
-      p[1] = x;
-    }
-  }
-
-  static inline void put_native_u4(address p, u4 x) {
-    switch ( intptr_t(p) & 3 ) {
-    case 0:  *(u4*)p = x;
-              break;
-
-    case 2:  ((u2*)p)[0] = x >> 16;
-             ((u2*)p)[1] = x;
-             break;
-
-    default: ((u1*)p)[0] = x >> 24;
-             ((u1*)p)[1] = x >> 16;
-             ((u1*)p)[2] = x >>  8;
-             ((u1*)p)[3] = x;
-             break;
-    }
-  }
-
-  static inline void put_native_u8(address p, u8 x) {
-    switch ( intptr_t(p) & 7 ) {
-    case 0:  *(u8*)p = x;
-             break;
-
-    case 4:  ((u4*)p)[0] = x >> 32;
-             ((u4*)p)[1] = x;
-             break;
-
-    case 2:  ((u2*)p)[0] = x >> 48;
-             ((u2*)p)[1] = x >> 32;
-             ((u2*)p)[2] = x >> 16;
-             ((u2*)p)[3] = x;
-             break;
-
-    default: ((u1*)p)[0] = x >> 56;
-             ((u1*)p)[1] = x >> 48;
-             ((u1*)p)[2] = x >> 40;
-             ((u1*)p)[3] = x >> 32;
-             ((u1*)p)[4] = x >> 24;
-             ((u1*)p)[5] = x >> 16;
-             ((u1*)p)[6] = x >>  8;
-             ((u1*)p)[7] = x;
-    }
-  }
-
-  // Efficient reading and writing of unaligned unsigned data in Java byte ordering (i.e. big-endian ordering)
-  // (no byte-order reversal is needed since Power CPUs are big-endian oriented).
-  static inline u2   get_Java_u2(address p) { return get_native_u2(p); }
-  static inline u4   get_Java_u4(address p) { return get_native_u4(p); }
-  static inline u8   get_Java_u8(address p) { return get_native_u8(p); }
-
-  static inline void put_Java_u2(address p, u2 x)     { put_native_u2(p, x); }
-  static inline void put_Java_u4(address p, u4 x)     { put_native_u4(p, x); }
-  static inline void put_Java_u8(address p, u8 x)     { put_native_u8(p, x); }
-
-#endif // VM_LITTLE_ENDIAN
-};
-
-#endif // CPU_PPC_BYTES_PPC_HPP

src/hotspot/cpu/ppc/c1_LIRAssembler_ppc.cpp

@@ -2250,7 +2250,7 @@ void LIR_Assembler::emit_alloc_obj(LIR_OpAllocObj* op) {
 
 void LIR_Assembler::emit_alloc_array(LIR_OpAllocArray* op) {
   LP64_ONLY( __ extsw(op->len()->as_register(), op->len()->as_register()); )
-  if (UseSlowPath ||
+  if (UseSlowPath || op->always_slow_path() ||
       (!UseFastNewObjectArray && (is_reference_type(op->type()))) ||
       (!UseFastNewTypeArray   && (!is_reference_type(op->type())))) {
     __ b(*op->stub()->entry());
@@ -3180,13 +3180,9 @@ void LIR_Assembler::emit_opSubstitutabilityCheck(LIR_OpSubstitutabilityCheck* op
   } else {
     Register tmp1 = op->tmp1()->as_register();
     Register tmp2 = op->tmp2()->as_register();
-    if (left == right) { // same operand, so clearly the same klasses, let's save the check
-      __ b(*op->stub()->entry());  //  -> do slow check
-    } else {
-      __ cmp_klasses_from_objects(CR0, left, right, tmp1, tmp2);
-      __ bc_far_optimized(Assembler::bcondCRbiIs1, __ bi0(CR0, Assembler::equal),
-                          *op->stub()->entry()); // same klass -> do slow check
-    }
+    __ cmp_klasses_from_objects(CR0, left, right, tmp1, tmp2);
+    __ bc_far_optimized(Assembler::bcondCRbiIs1, __ bi0(CR0, Assembler::equal),
+                        *op->stub()->entry()); // same klass -> do slow check
     // fall through to L_oops_not_equal
   }
 

src/hotspot/cpu/ppc/downcallLinker_ppc.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2020, 2025 SAP SE. All rights reserved.
+ * Copyright (c) 2020, 2026 SAP SE. All rights reserved.
  * Copyright (c) 2020, 2026, Oracle and/or its affiliates. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
@@ -297,7 +297,7 @@ void DowncallLinker::StubGenerator::generate() {
   Label L_after_reguard;
 
   if (_needs_transition) {
-    __ li(tmp, _thread_in_native_trans);
+    __ li(tmp, _thread_in_vm);
     __ release();
     __ stw(tmp, in_bytes(JavaThread::thread_state_offset()), R16_thread);
     if (!UseSystemMemoryBarrier) {
@@ -374,5 +374,5 @@ void DowncallLinker::StubGenerator::generate() {
 
   //////////////////////////////////////////////////////////////////////////////
 
-  __ flush();
+  // Code will be copied. No ICache sync required.
 }

src/hotspot/cpu/ppc/frame_ppc.cpp

@@ -505,8 +505,8 @@ intptr_t* frame::repair_sender_sp(nmethod* nm, intptr_t* sp, intptr_t** saved_fp
 }
 
 bool frame::was_augmented_on_entry(int& real_size) const {
-  assert(is_compiled_frame(), "");
-  if (_cb->as_nmethod_or_null()->needs_stack_repair()) {
+  assert(_cb != nullptr && _cb->is_nmethod(), "");
+  if (_cb->as_nmethod()->needs_stack_repair()) {
     Unimplemented();
   }
   real_size = _cb->frame_size();

src/hotspot/cpu/ppc/interp_masm_ppc_64.cpp

@@ -2450,7 +2450,7 @@ void InterpreterMacroAssembler::write_flat_field(Register entry, Register tmp1,
   payload_address(value, value, tmp1, tmp2);
 
   Register layout_info = field_offset;
-  lbz(tmp1, in_bytes(ResolvedFieldEntry::field_index_offset()), entry);
+  lhz(tmp1, in_bytes(ResolvedFieldEntry::field_index_offset()), entry);
   ld(tmp2, in_bytes(ResolvedFieldEntry::field_holder_offset()), entry);
   inline_layout_info(tmp2, tmp1, layout_info);
 

src/hotspot/cpu/ppc/interpreterRT_ppc.cpp

@@ -1,6 +1,6 @@
 /*
- * Copyright (c) 1997, 2025, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2012, 2025 SAP SE. All rights reserved.
+ * Copyright (c) 1997, 2026, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2026 SAP SE. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * This code is free software; you can redistribute it and/or modify it
@@ -127,7 +127,7 @@ void InterpreterRuntime::SignatureHandlerGenerator::generate(uint64_t fingerprin
   __ load_const(R3_RET, AbstractInterpreter::result_handler(method()->result_type()));
   __ blr();
 
-  __ flush();
+  __ invalidate_icache();
 }
 
 #undef __

src/hotspot/cpu/ppc/jniFastGetField_ppc.cpp

@@ -99,7 +99,7 @@ address JNI_FastGetField::generate_fast_get_int_field0(BasicType type) {
   BarrierSetAssembler* bs = BarrierSet::barrier_set()->barrier_set_assembler();
   bs->try_resolve_jobject_in_native(masm, Robj, R3_ARG1, R4_ARG2, Rtmp, slow);
 
-  __ srwi(Rtmp, R5_ARG3, jfieldIDWorkaround::offset_shift); // offset
+  __ srdi(Rtmp, R5_ARG3, jfieldIDWorkaround::offset_shift); // offset
 
   assert(count < LIST_CAPACITY, "LIST_CAPACITY too small");
   speculative_load_pclist[count] = __ pc();   // Used by the segfault handler
@@ -154,7 +154,7 @@ address JNI_FastGetField::generate_fast_get_int_field0(BasicType type) {
   __ load_const_optimized(R12, slow_case_addr, R0);
   __ call_c_and_return_to_caller(R12); // tail call
 
-  __ flush();
+  __ invalidate_icache();
 
   return fast_entry;
 }

src/hotspot/cpu/ppc/macroAssembler_ppc.cpp

@@ -2688,7 +2688,6 @@ void MacroAssembler::tlab_allocate(
 void MacroAssembler::compiler_fast_lock_object(ConditionRegister flag, Register obj, Register box,
                                                Register tmp1, Register tmp2, Register tmp3) {
   assert_different_registers(obj, box, tmp1, tmp2, tmp3);
-  assert(UseObjectMonitorTable || tmp3 == noreg, "tmp3 not needed");
   assert(flag == CR0, "bad condition register");
 
   // Handle inflated monitor.
@@ -2698,11 +2697,9 @@ void MacroAssembler::compiler_fast_lock_object(ConditionRegister flag, Register
   // Finish fast lock unsuccessfully. MUST branch to with flag == EQ
   Label slow_path;
 
-  if (UseObjectMonitorTable) {
-    // Clear cache in case fast locking succeeds or we need to take the slow-path.
-    li(tmp1, 0);
-    std(tmp1, in_bytes(BasicObjectLock::lock_offset()) + BasicLock::object_monitor_cache_offset_in_bytes(), box);
-  }
+  // Clear cache in case fast locking succeeds or we need to take the slow-path.
+  li(tmp1, 0);
+  std(tmp1, in_bytes(BasicObjectLock::lock_offset()) + BasicLock::object_monitor_cache_offset_in_bytes(), box);
 
   if (DiagnoseSyncOnValueBasedClasses != 0) {
     load_klass(tmp1, obj);
@@ -2760,9 +2757,9 @@ void MacroAssembler::compiler_fast_lock_object(ConditionRegister flag, Register
 
     // mark contains the tagged ObjectMonitor*.
     const uintptr_t monitor_tag = markWord::monitor_value;
-    const Register monitor    = UseObjectMonitorTable ? tmp1 : noreg;
+    const Register monitor    = tmp1;
     const Register owner_addr = tmp2;
-    const Register thread_id  = UseObjectMonitorTable ? tmp3 : tmp1;
+    const Register thread_id  = tmp3;
     // Offsets into the current thread's object monitor cache (omc).
     const ByteSize thr_omc_offset     = JavaThread::om_cache_offset();
     const ByteSize omc_monitor_offset = OMCache::monitor_offset();
@@ -2770,61 +2767,55 @@ void MacroAssembler::compiler_fast_lock_object(ConditionRegister flag, Register
 
     Label monitor_locked;
 
-    if (!UseObjectMonitorTable) {
-      // Compute owner address.
-      addi(owner_addr, mark, in_bytes(ObjectMonitor::owner_offset()) - monitor_tag);
-      mark = noreg;
-    } else {
-      const Register tmp3_bucket = tmp3;
-      const Register tmp2_hash = tmp2;
-      Label monitor_found;
+    const Register tmp3_bucket = tmp3;
+    const Register tmp2_hash = tmp2;
+    Label monitor_found;
 
-      // Save the mark, we might need it to extract the hash.
-      mr(tmp2_hash, mark);
+    // Save the mark, we might need it to extract the hash.
+    mr(tmp2_hash, mark);
 
-      // Look for the monitor in the current thread's object monitor cache (omc).
+    // Look for the monitor in the current thread's object monitor cache (omc).
 
-      ld(R0, in_bytes(thr_omc_offset + omc_obj_offset), R16_thread);
-      ld(monitor, in_bytes(thr_omc_offset + omc_monitor_offset), R16_thread);
-      cmpd(CR0, R0, obj);
-      beq(CR0, monitor_found);
+    ld(R0, in_bytes(thr_omc_offset + omc_obj_offset), R16_thread);
+    ld(monitor, in_bytes(thr_omc_offset + omc_monitor_offset), R16_thread);
+    cmpd(CR0, R0, obj);
+    beq(CR0, monitor_found);
 
-      // Look for the monitor in the table.
+    // Look for the monitor in the table.
 
-      // Get the hash code.
-      srdi(tmp2_hash, tmp2_hash, markWord::hash_shift);
+    // Get the hash code.
+    srdi(tmp2_hash, tmp2_hash, markWord::hash_shift);
 
-      // Get the table and calculate the bucket's address
-      int simm16_rest = load_const_optimized(tmp3, ObjectMonitorTable::current_table_address(), R0, true);
-      ld_ptr(tmp3, simm16_rest, tmp3);
-      ld(tmp1, in_bytes(ObjectMonitorTable::table_capacity_mask_offset()), tmp3);
-      andr(tmp2_hash, tmp2_hash, tmp1);
-      ld(tmp3_bucket, in_bytes(ObjectMonitorTable::table_buckets_offset()), tmp3);
+    // Get the table and calculate the bucket's address
+    int simm16_rest = load_const_optimized(tmp3, ObjectMonitorTable::current_table_address(), R0, true);
+    ld_ptr(tmp3, simm16_rest, tmp3);
+    ld(tmp1, in_bytes(ObjectMonitorTable::table_capacity_mask_offset()), tmp3);
+    andr(tmp2_hash, tmp2_hash, tmp1);
+    ld(tmp3_bucket, in_bytes(ObjectMonitorTable::table_buckets_offset()), tmp3);
 
-      // Read the monitor from the bucket.
-      sldi(tmp2_hash, tmp2_hash, LogBytesPerWord);
-      ldx(monitor, tmp3_bucket, tmp2_hash);
+    // Read the monitor from the bucket.
+    sldi(tmp2_hash, tmp2_hash, LogBytesPerWord);
+    ldx(monitor, tmp3_bucket, tmp2_hash);
 
-      // Check if the monitor in the bucket is special (empty, tombstone or removed).
-      cmpldi(CR0, monitor, ObjectMonitorTable::SpecialPointerValues::below_is_special);
-      blt(CR0, slow_path);
+    // Check if the monitor in the bucket is special (empty, tombstone or removed).
+    cmpldi(CR0, monitor, ObjectMonitorTable::SpecialPointerValues::below_is_special);
+    blt(CR0, slow_path);
 
-      // Check if object matches.
-      ld(tmp3, in_bytes(ObjectMonitor::object_offset()), monitor);
-      BarrierSetAssembler* bs_asm = BarrierSet::barrier_set()->barrier_set_assembler();
-      bs_asm->try_peek_weak_handle_in_nmethod(this, tmp3, tmp3, tmp2, slow_path);
-      cmpd(CR0, tmp3, obj);
-      bne(CR0, slow_path);
+    // Check if object matches.
+    ld(tmp3, in_bytes(ObjectMonitor::object_offset()), monitor);
+    BarrierSetAssembler* bs_asm = BarrierSet::barrier_set()->barrier_set_assembler();
+    bs_asm->try_peek_weak_handle_in_nmethod(this, tmp3, tmp3, tmp2, slow_path);
+    cmpd(CR0, tmp3, obj);
+    bne(CR0, slow_path);
 
-      // Store the monitor in the current thread's object monitor cache (omc).
-      std(monitor, in_bytes(thr_omc_offset + omc_monitor_offset), R16_thread);
-      std(obj, in_bytes(thr_omc_offset + omc_obj_offset), R16_thread);
+    // Store the monitor in the current thread's object monitor cache (omc).
+    std(monitor, in_bytes(thr_omc_offset + omc_monitor_offset), R16_thread);
+    std(obj, in_bytes(thr_omc_offset + omc_obj_offset), R16_thread);
 
-      bind(monitor_found);
+    bind(monitor_found);
 
-      // Compute owner address.
-      addi(owner_addr, monitor, in_bytes(ObjectMonitor::owner_offset()));
-    }
+    // Compute owner address.
+    addi(owner_addr, monitor, in_bytes(ObjectMonitor::owner_offset()));
 
     // Try to CAS owner (no owner => current thread's _monitor_owner_id).
     assert_different_registers(thread_id, monitor, owner_addr, box, R0);
@@ -2843,23 +2834,14 @@ void MacroAssembler::compiler_fast_lock_object(ConditionRegister flag, Register
     bne(CR0, slow_path);
 
     // Recursive.
-    if (!UseObjectMonitorTable) {
-      assert_different_registers(tmp1, owner_addr);
-      ld(tmp1, in_bytes(ObjectMonitor::recursions_offset() - ObjectMonitor::owner_offset()), owner_addr);
-      addi(tmp1, tmp1, 1);
-      std(tmp1, in_bytes(ObjectMonitor::recursions_offset() - ObjectMonitor::owner_offset()), owner_addr);
-    } else {
-      assert_different_registers(tmp2, monitor);
-      ld(tmp2, in_bytes(ObjectMonitor::recursions_offset()), monitor);
-      addi(tmp2, tmp2, 1);
-      std(tmp2, in_bytes(ObjectMonitor::recursions_offset()), monitor);
-    }
+    assert_different_registers(tmp2, monitor);
+    ld(tmp2, in_bytes(ObjectMonitor::recursions_offset()), monitor);
+    addi(tmp2, tmp2, 1);
+    std(tmp2, in_bytes(ObjectMonitor::recursions_offset()), monitor);
 
     bind(monitor_locked);
-    if (UseObjectMonitorTable) {
-      // Cache the monitor for unlock.
-      std(monitor, BasicLock::object_monitor_cache_offset_in_bytes(), box);
-    }
+    // Cache the monitor for unlock.
+    std(monitor, BasicLock::object_monitor_cache_offset_in_bytes(), box);
   }
 
   bind(locked);
@@ -2926,11 +2908,7 @@ void MacroAssembler::compiler_fast_unlock_object(ConditionRegister flag, Registe
     // Check for monitor (0b10).
     ld(mark, oopDesc::mark_offset_in_bytes(), obj);
     andi_(t, mark, markWord::monitor_value);
-    if (!UseObjectMonitorTable) {
-      bne(CR0, inflated);
-    } else {
-      bne(CR0, push_and_slow);
-    }
+    bne(CR0, push_and_slow);
 
 #ifdef ASSERT
     // Check header not unlocked (0b01).
@@ -2980,15 +2958,10 @@ void MacroAssembler::compiler_fast_unlock_object(ConditionRegister flag, Registe
     const Register monitor = mark;
     const uintptr_t monitor_tag = markWord::monitor_value;
 
-    if (!UseObjectMonitorTable) {
-      // Untag the monitor.
-      subi(monitor, mark, monitor_tag);
-    } else {
-      ld(monitor, BasicLock::object_monitor_cache_offset_in_bytes(), box);
-      // null check with Flags == NE, no valid pointer below alignof(ObjectMonitor*)
-      cmpldi(CR0, monitor, checked_cast<uint8_t>(alignof(ObjectMonitor*)));
-      blt(CR0, slow_path);
-    }
+    ld(monitor, BasicLock::object_monitor_cache_offset_in_bytes(), box);
+    // null check with Flags == NE, no valid pointer below alignof(ObjectMonitor*)
+    cmpldi(CR0, monitor, checked_cast<uint8_t>(alignof(ObjectMonitor*)));
+    blt(CR0, slow_path);
 
     const Register recursions = tmp2;
     Label not_recursive;
@@ -3359,16 +3332,6 @@ void MacroAssembler::test_oop_prototype_bit(Register oop, Register temp_reg, int
                                             Label& jmp_label, bool maybe_far) {
   // load mark word
   ld(temp_reg, oopDesc::mark_offset_in_bytes(), oop);
-  if (!UseObjectMonitorTable) {
-    Label test_mark_word;
-    // if unlocked bit is set we can directly use the mark word
-    andi_(R0, temp_reg, markWord::unlocked_value);
-    bne(CR0, test_mark_word);
-    // slow path use klass prototype
-    load_prototype_header(temp_reg, oop);
-
-    bind(test_mark_word);
-  }
   andi_(R0, temp_reg, test_bit);
   if (maybe_far) {
     bc_far_optimized(jmp_set ? Assembler::bcondCRbiIs0 : Assembler::bcondCRbiIs1,
@@ -4907,11 +4870,9 @@ void MacroAssembler::fast_lock(Register box, Register obj, Register t1, Register
   Label push;
   const Register t = R0;
 
-  if (UseObjectMonitorTable) {
-    // Clear cache in case fast locking succeeds or we need to take the slow-path.
-    li(t, 0);
-    std(t, in_bytes(BasicObjectLock::lock_offset()) + BasicLock::object_monitor_cache_offset_in_bytes(), box);
-  }
+  // Clear cache in case fast locking succeeds or we need to take the slow-path.
+  li(t, 0);
+  std(t, in_bytes(BasicObjectLock::lock_offset()) + BasicLock::object_monitor_cache_offset_in_bytes(), box);
 
   if (DiagnoseSyncOnValueBasedClasses != 0) {
     load_klass(t1, obj);

src/hotspot/cpu/ppc/macroAssembler_ppc_sha.cpp

@@ -613,7 +613,6 @@ void MacroAssembler::sha512_update_sha_state(const Register state,
   VectorRegister ini_e = VR14;
   VectorRegister ini_g = VR16;
   static const VectorRegister inis[] = {ini_a, ini_c, ini_e, ini_g};
-  static const int total_inis = sizeof(inis)/sizeof(VectorRegister);
 
   Label state_save_aligned, after_state_save_aligned;
 

src/hotspot/cpu/ppc/ppc.ad

@@ -10985,24 +10985,7 @@ instruct partialSubtypeCheckConstSuper(rarg3RegP sub, rarg2RegP super_reg, immP
 
 // inlined locking and unlocking
 
-instruct cmpFastLock(flagsRegCR0 crx, iRegPdst oop, iRegPdst box, iRegPdst tmp1, iRegPdst tmp2) %{
-  predicate(!UseObjectMonitorTable);
-  match(Set crx (FastLock oop box));
-  effect(TEMP tmp1, TEMP tmp2);
-
-  format %{ "FASTLOCK  $oop, $box, $tmp1, $tmp2" %}
-  ins_encode %{
-    __ fast_lock($crx$$CondRegister, $oop$$Register, $box$$Register,
-                 $tmp1$$Register, $tmp2$$Register, noreg /*tmp3*/);
-    // If locking was successful, crx should indicate 'EQ'.
-    // The compiler generates a branch to the runtime call to
-    // _complete_monitor_locking_Java for the case where crx is 'NE'.
-  %}
-  ins_pipe(pipe_class_compare);
-%}
-
-instruct cmpFastLockMonitorTable(flagsRegCR0 crx, iRegPdst oop, iRegPdst box, iRegPdst tmp1, iRegPdst tmp2, iRegPdst tmp3, flagsRegCR1 cr1) %{
-  predicate(UseObjectMonitorTable);
+instruct cmpFastLock(flagsRegCR0 crx, iRegPdst oop, iRegPdst box, iRegPdst tmp1, iRegPdst tmp2, iRegPdst tmp3, flagsRegCR1 cr1) %{
   match(Set crx (FastLock oop box));
   effect(TEMP tmp1, TEMP tmp2, TEMP tmp3, KILL cr1);
 

src/hotspot/cpu/ppc/runtime_ppc.cpp

@@ -1,6 +1,6 @@
 /*
- * Copyright (c) 1998, 2025, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2012, 2025 SAP SE. All rights reserved.
+ * Copyright (c) 1998, 2026, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2026 SAP SE. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * This code is free software; you can redistribute it and/or modify it
@@ -141,8 +141,7 @@ ExceptionBlob* OptoRuntime::generate_exception_blob() {
   __ mtlr(R4_ARG2);
   __ bctr();
 
-  // Make sure all code is generated.
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   // Set exception blob.
   return ExceptionBlob::create(&buffer, oop_maps,

src/hotspot/cpu/ppc/sharedRuntime_ppc.cpp

@@ -2169,7 +2169,7 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
     assert(vep_offset != -1,        "Must be set");
 #endif
 
-    __ flush();
+    // Code will be copied. No ICache sync required.
     nmethod* nm = nmethod::new_native_nmethod(method,
                                               compile_id,
                                               masm->code(),
@@ -2198,7 +2198,7 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
                          in_sig_bt,
                          in_regs);
     int frame_complete = ((intptr_t)__ pc()) - start;  // not complete, period
-    __ flush();
+    // Code will be copied. No ICache sync required.
     int stack_slots = SharedRuntime::out_preserve_stack_slots();  // no out slots at all, actually
     return nmethod::new_native_nmethod(method,
                                        compile_id,
@@ -2525,8 +2525,7 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
 
     // Try fastpath for locking.
     // fast_lock kills r_temp_1, r_temp_2, r_temp_3.
-    Register r_temp_3_or_noreg = UseObjectMonitorTable ? r_temp_3 : noreg;
-    __ compiler_fast_lock_object(CR0, r_oop, r_box, r_temp_1, r_temp_2, r_temp_3_or_noreg);
+    __ compiler_fast_lock_object(CR0, r_oop, r_box, r_temp_1, r_temp_2, r_temp_3);
     __ beq(CR0, locked);
 
     // None of the above fast optimizations worked so we have to get into the
@@ -2618,21 +2617,8 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
   }
 
   // Publish thread state
-  // --------------------------------------------------------------------------
-
-  // Switch thread to "native transition" state before reading the
-  // synchronization state. This additional state is necessary because reading
-  // and testing the synchronization state is not atomic w.r.t. GC, as this
-  // scenario demonstrates:
-  //   - Java thread A, in _thread_in_native state, loads _not_synchronized
-  //     and is preempted.
-  //   - VM thread changes sync state to synchronizing and suspends threads
-  //     for GC.
-  //   - Thread A is resumed to finish this native method, but doesn't block
-  //     here since it didn't see any synchronization in progress, and escapes.
-
-  // Transition from _thread_in_native to _thread_in_native_trans.
-  __ li(R0, _thread_in_native_trans);
+  // Transition from _thread_in_native to _thread_in_vm.
+  __ li(R0, _thread_in_vm);
   __ release();
   // TODO: PPC port assert(4 == JavaThread::sz_thread_state(), "unexpected field size");
   __ stw(R0, thread_(thread_state));
@@ -2683,10 +2669,10 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
     // Publish thread state.
     // --------------------------------------------------------------------------
 
-    // Thread state is thread_in_native_trans. Any safepoint blocking has
+    // Thread state is _thread_in_vm. Any safepoint blocking has
     // already happened so we can now change state to _thread_in_Java.
 
-    // Transition from _thread_in_native_trans to _thread_in_Java.
+    // Transition from _thread_in_vm to _thread_in_Java.
     __ li(R0, _thread_in_Java);
     __ lwsync(); // Acquire safepoint and suspend state, release thread state.
     // TODO: PPC port assert(4 == JavaThread::sz_thread_state(), "unexpected field size");
@@ -2850,7 +2836,7 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
   // Done.
   // --------------------------------------------------------------------------
 
-  __ flush();
+  // Code will be copied. No ICache sync required.
 
   nmethod *nm = nmethod::new_native_nmethod(method,
                                             compile_id,
@@ -3217,8 +3203,7 @@ void SharedRuntime::generate_deopt_blob() {
   __ unimplemented("deopt blob needed only with compiler");
 #endif
 
-  // Make sure all code is generated
-  __ flush();
+  // Code will be copied. No ICache sync required.
 
   _deopt_blob = DeoptimizationBlob::create(&buffer, oop_maps, 0, exception_offset,
                                            reexecute_offset, first_frame_size_in_bytes / wordSize);
@@ -3355,7 +3340,7 @@ UncommonTrapBlob* OptoRuntime::generate_uncommon_trap_blob() {
   // Return to the interpreter entry point.
   __ blr();
 
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   return UncommonTrapBlob::create(&buffer, oop_maps, frame_size_in_bytes/wordSize);
 }
@@ -3461,8 +3446,7 @@ SafepointBlob* SharedRuntime::generate_handler_blob(StubId id, address call_ptr)
 
   __ blr();
 
-  // Make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   // Fill-out other meta info
   // CodeBlob frame size is in words.
@@ -3548,9 +3532,7 @@ RuntimeStub* SharedRuntime::generate_resolve_blob(StubId id, address destination
   __ std(R11_scratch1, in_bytes(JavaThread::vm_result_oop_offset()), R16_thread);
   __ b64_patchable(StubRoutines::forward_exception_entry(), relocInfo::runtime_call_type);
 
-  // -------------
-  // Make sure all code is generated.
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   // return the blob
   // frame_size_words or bytes??

src/hotspot/cpu/ppc/stubGenerator_ppc.cpp

@@ -2781,10 +2781,8 @@ class StubGenerator: public StubCodeGenerator {
     Register to             = R4_ARG2;  // destination array address
     Register key            = R5_ARG3;  // round key array
 
-    Register keylen         = R8;
-    Register temp           = R9;
-    Register keypos         = R10;
-    Register fifteen        = R12;
+    Register keylen         = R6;
+    Register tmp            = R7;
 
     VectorRegister vRet     = VR0;
 
@@ -2793,68 +2791,27 @@ class StubGenerator: public StubCodeGenerator {
     VectorRegister vKey3    = VR3;
     VectorRegister vKey4    = VR4;
 
-    VectorRegister fromPerm = VR5;
-    VectorRegister keyPerm  = VR6;
-    VectorRegister toPerm   = VR7;
-    VectorRegister fSplt    = VR8;
+    VectorRegister vp       = VR6;  // permute vector for byte vector accesses on P8 LE
 
-    VectorRegister vTmp1    = VR9;
-    VectorRegister vTmp2    = VR10;
-    VectorRegister vTmp3    = VR11;
-    VectorRegister vTmp4    = VR12;
-
-    __ li              (fifteen, 15);
+    __ compute_vp_for_byte_vector_unaligned(vp, /*temp*/ vRet);
 
     // load unaligned from[0-15] to vRet
-    __ lvx             (vRet, from);
-    __ lvx             (vTmp1, fifteen, from);
-    __ lvsl            (fromPerm, from);
-#ifdef VM_LITTLE_ENDIAN
-    __ vspltisb        (fSplt, 0x0f);
-    __ vxor            (fromPerm, fromPerm, fSplt);
-#endif
-    __ vperm           (vRet, vRet, vTmp1, fromPerm);
+    __ load_byte_vector_unaligned(vRet, 0, from, tmp, vp);
+
+    // load the 1st round key to vKey1
+    __ load_word_vector_unaligned(vKey1, 0, key, tmp);
 
     // load keylen (44 or 52 or 60)
     __ lwz             (keylen, arrayOopDesc::length_offset_in_bytes() - arrayOopDesc::base_offset_in_bytes(T_INT), key);
 
-    // to load keys
-    __ load_perm       (keyPerm, key);
-#ifdef VM_LITTLE_ENDIAN
-    __ vspltisb        (vTmp2, -16);
-    __ vrld            (keyPerm, keyPerm, vTmp2);
-    __ vrld            (keyPerm, keyPerm, vTmp2);
-    __ vsldoi          (keyPerm, keyPerm, keyPerm, 8);
-#endif
-
-    // load the 1st round key to vTmp1
-    __ lvx             (vTmp1, key);
-    __ li              (keypos, 16);
-    __ lvx             (vKey1, keypos, key);
-    __ vec_perm        (vTmp1, vKey1, keyPerm);
-
     // 1st round
-    __ vxor            (vRet, vRet, vTmp1);
+    __ vxor            (vRet, vRet, vKey1);
 
-    // load the 2nd round key to vKey1
-    __ li              (keypos, 32);
-    __ lvx             (vKey2, keypos, key);
-    __ vec_perm        (vKey1, vKey2, keyPerm);
-
-    // load the 3rd round key to vKey2
-    __ li              (keypos, 48);
-    __ lvx             (vKey3, keypos, key);
-    __ vec_perm        (vKey2, vKey3, keyPerm);
-
-    // load the 4th round key to vKey3
-    __ li              (keypos, 64);
-    __ lvx             (vKey4, keypos, key);
-    __ vec_perm        (vKey3, vKey4, keyPerm);
-
-    // load the 5th round key to vKey4
-    __ li              (keypos, 80);
-    __ lvx             (vTmp1, keypos, key);
-    __ vec_perm        (vKey4, vTmp1, keyPerm);
+    // load the 2nd - 5th round key to vKey1 - vKey4
+    __ load_word_vector_unaligned(vKey1, 16, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 32, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 48, key, tmp);
+    __ load_word_vector_unaligned(vKey4, 64, key, tmp);
 
     // 2nd - 5th rounds
     __ vcipher         (vRet, vRet, vKey1);
@@ -2862,25 +2819,11 @@ class StubGenerator: public StubCodeGenerator {
     __ vcipher         (vRet, vRet, vKey3);
     __ vcipher         (vRet, vRet, vKey4);
 
-    // load the 6th round key to vKey1
-    __ li              (keypos, 96);
-    __ lvx             (vKey2, keypos, key);
-    __ vec_perm        (vKey1, vTmp1, vKey2, keyPerm);
-
-    // load the 7th round key to vKey2
-    __ li              (keypos, 112);
-    __ lvx             (vKey3, keypos, key);
-    __ vec_perm        (vKey2, vKey3, keyPerm);
-
-    // load the 8th round key to vKey3
-    __ li              (keypos, 128);
-    __ lvx             (vKey4, keypos, key);
-    __ vec_perm        (vKey3, vKey4, keyPerm);
-
-    // load the 9th round key to vKey4
-    __ li              (keypos, 144);
-    __ lvx             (vTmp1, keypos, key);
-    __ vec_perm        (vKey4, vTmp1, keyPerm);
+    // load the 6th - 9th round key to vKey1 - vKey4
+    __ load_word_vector_unaligned(vKey1, 80, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 96, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 112, key, tmp);
+    __ load_word_vector_unaligned(vKey4, 128, key, tmp);
 
     // 6th - 9th rounds
     __ vcipher         (vRet, vRet, vKey1);
@@ -2888,15 +2831,9 @@ class StubGenerator: public StubCodeGenerator {
     __ vcipher         (vRet, vRet, vKey3);
     __ vcipher         (vRet, vRet, vKey4);
 
-    // load the 10th round key to vKey1
-    __ li              (keypos, 160);
-    __ lvx             (vKey2, keypos, key);
-    __ vec_perm        (vKey1, vTmp1, vKey2, keyPerm);
-
-    // load the 11th round key to vKey2
-    __ li              (keypos, 176);
-    __ lvx             (vTmp1, keypos, key);
-    __ vec_perm        (vKey2, vTmp1, keyPerm);
+    // load the 10th - 11th round key to vKey1 - vKey2
+    __ load_word_vector_unaligned(vKey1, 144, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 160, key, tmp);
 
     // if all round keys are loaded, skip next 4 rounds
     __ cmpwi           (CR0, keylen, 44);
@@ -2906,15 +2843,9 @@ class StubGenerator: public StubCodeGenerator {
     __ vcipher         (vRet, vRet, vKey1);
     __ vcipher         (vRet, vRet, vKey2);
 
-    // load the 12th round key to vKey1
-    __ li              (keypos, 192);
-    __ lvx             (vKey2, keypos, key);
-    __ vec_perm        (vKey1, vTmp1, vKey2, keyPerm);
-
-    // load the 13th round key to vKey2
-    __ li              (keypos, 208);
-    __ lvx             (vTmp1, keypos, key);
-    __ vec_perm        (vKey2, vTmp1, keyPerm);
+    // load the 12th - 13th round key to vKey1 - vKey2
+    __ load_word_vector_unaligned(vKey1, 176, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 192, key, tmp);
 
     // if all round keys are loaded, skip next 2 rounds
     __ cmpwi           (CR0, keylen, 52);
@@ -2929,15 +2860,9 @@ class StubGenerator: public StubCodeGenerator {
     __ vcipher         (vRet, vRet, vKey1);
     __ vcipher         (vRet, vRet, vKey2);
 
-    // load the 14th round key to vKey1
-    __ li              (keypos, 224);
-    __ lvx             (vKey2, keypos, key);
-    __ vec_perm        (vKey1, vTmp1, vKey2, keyPerm);
-
-    // load the 15th round key to vKey2
-    __ li              (keypos, 240);
-    __ lvx             (vTmp1, keypos, key);
-    __ vec_perm        (vKey2, vTmp1, keyPerm);
+    // load the 14th - 15th round key to vKey1 - vKey2
+    __ load_word_vector_unaligned(vKey1, 208, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 224, key, tmp);
 
     __ bind(L_doLast);
 
@@ -2945,23 +2870,8 @@ class StubGenerator: public StubCodeGenerator {
     __ vcipher         (vRet, vRet, vKey1);
     __ vcipherlast     (vRet, vRet, vKey2);
 
-#ifdef VM_LITTLE_ENDIAN
-    // toPerm = 0x0F0E0D0C0B0A09080706050403020100
-    __ lvsl            (toPerm, keypos); // keypos is a multiple of 16
-    __ vxor            (toPerm, toPerm, fSplt);
-
-    // Swap Bytes
-    __ vperm           (vRet, vRet, vRet, toPerm);
-#endif
-
     // store result (unaligned)
-    // Note: We can't use a read-modify-write sequence which touches additional Bytes.
-    Register lo = temp, hi = fifteen; // Reuse
-    __ vsldoi          (vTmp1, vRet, vRet, 8);
-    __ mfvrd           (hi, vRet);
-    __ mfvrd           (lo, vTmp1);
-    __ std             (hi, 0 LITTLE_ENDIAN_ONLY(+ 8), to);
-    __ std             (lo, 0 BIG_ENDIAN_ONLY(+ 8), to);
+    __ store_byte_vector_unaligned(vRet, 0, to, tmp, vp);
 
     __ blr();
 
@@ -2989,10 +2899,8 @@ class StubGenerator: public StubCodeGenerator {
     Register to             = R4_ARG2;  // destination array address
     Register key            = R5_ARG3;  // round key array
 
-    Register keylen         = R8;
-    Register temp           = R9;
-    Register keypos         = R10;
-    Register fifteen        = R12;
+    Register keylen         = R6;
+    Register tmp            = R7;
 
     VectorRegister vRet     = VR0;
 
@@ -3002,41 +2910,16 @@ class StubGenerator: public StubCodeGenerator {
     VectorRegister vKey4    = VR4;
     VectorRegister vKey5    = VR5;
 
-    VectorRegister fromPerm = VR6;
-    VectorRegister keyPerm  = VR7;
-    VectorRegister toPerm   = VR8;
-    VectorRegister fSplt    = VR9;
+    VectorRegister vp       = VR6;  // permute vector for byte vector accesses on P8 LE
 
-    VectorRegister vTmp1    = VR10;
-    VectorRegister vTmp2    = VR11;
-    VectorRegister vTmp3    = VR12;
-    VectorRegister vTmp4    = VR13;
-
-    __ li              (fifteen, 15);
+    __ compute_vp_for_byte_vector_unaligned(vp, /*temp*/ vRet);
 
     // load unaligned from[0-15] to vRet
-    __ lvx             (vRet, from);
-    __ lvx             (vTmp1, fifteen, from);
-    __ lvsl            (fromPerm, from);
-#ifdef VM_LITTLE_ENDIAN
-    __ vspltisb        (fSplt, 0x0f);
-    __ vxor            (fromPerm, fromPerm, fSplt);
-#endif
-    __ vperm           (vRet, vRet, vTmp1, fromPerm); // align [and byte swap in LE]
+    __ load_byte_vector_unaligned(vRet, 0, from, tmp, vp);
 
     // load keylen (44 or 52 or 60)
     __ lwz             (keylen, arrayOopDesc::length_offset_in_bytes() - arrayOopDesc::base_offset_in_bytes(T_INT), key);
 
-    // to load keys
-    __ load_perm       (keyPerm, key);
-#ifdef VM_LITTLE_ENDIAN
-    __ vxor            (vTmp2, vTmp2, vTmp2);
-    __ vspltisb        (vTmp2, -16);
-    __ vrld            (keyPerm, keyPerm, vTmp2);
-    __ vrld            (keyPerm, keyPerm, vTmp2);
-    __ vsldoi          (keyPerm, keyPerm, keyPerm, 8);
-#endif
-
     __ cmpwi           (CR0, keylen, 44);
     __ beq             (CR0, L_do44);
 
@@ -3048,32 +2931,12 @@ class StubGenerator: public StubCodeGenerator {
     __ bne             (CR0, L_error);
 #endif
 
-    // load the 15th round key to vKey1
-    __ li              (keypos, 240);
-    __ lvx             (vKey1, keypos, key);
-    __ li              (keypos, 224);
-    __ lvx             (vKey2, keypos, key);
-    __ vec_perm        (vKey1, vKey2, vKey1, keyPerm);
-
-    // load the 14th round key to vKey2
-    __ li              (keypos, 208);
-    __ lvx             (vKey3, keypos, key);
-    __ vec_perm        (vKey2, vKey3, vKey2, keyPerm);
-
-    // load the 13th round key to vKey3
-    __ li              (keypos, 192);
-    __ lvx             (vKey4, keypos, key);
-    __ vec_perm        (vKey3, vKey4, vKey3, keyPerm);
-
-    // load the 12th round key to vKey4
-    __ li              (keypos, 176);
-    __ lvx             (vKey5, keypos, key);
-    __ vec_perm        (vKey4, vKey5, vKey4, keyPerm);
-
-    // load the 11th round key to vKey5
-    __ li              (keypos, 160);
-    __ lvx             (vTmp1, keypos, key);
-    __ vec_perm        (vKey5, vTmp1, vKey5, keyPerm);
+    // load the 15th - 11th round key to vKey1 - vKey5
+    __ load_word_vector_unaligned(vKey1, 224, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 208, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 192, key, tmp);
+    __ load_word_vector_unaligned(vKey4, 176, key, tmp);
+    __ load_word_vector_unaligned(vKey5, 160, key, tmp);
 
     // 1st - 5th rounds
     __ vxor            (vRet, vRet, vKey1);
@@ -3087,22 +2950,10 @@ class StubGenerator: public StubCodeGenerator {
     __ align(32);
     __ bind            (L_do52);
 
-    // load the 13th round key to vKey1
-    __ li              (keypos, 208);
-    __ lvx             (vKey1, keypos, key);
-    __ li              (keypos, 192);
-    __ lvx             (vKey2, keypos, key);
-    __ vec_perm        (vKey1, vKey2, vKey1, keyPerm);
-
-    // load the 12th round key to vKey2
-    __ li              (keypos, 176);
-    __ lvx             (vKey3, keypos, key);
-    __ vec_perm        (vKey2, vKey3, vKey2, keyPerm);
-
-    // load the 11th round key to vKey3
-    __ li              (keypos, 160);
-    __ lvx             (vTmp1, keypos, key);
-    __ vec_perm        (vKey3, vTmp1, vKey3, keyPerm);
+    // load the 13th - 11th round key to vKey1 - vKey3
+    __ load_word_vector_unaligned(vKey1, 192, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 176, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 160, key, tmp);
 
     // 1st - 3rd rounds
     __ vxor            (vRet, vRet, vKey1);
@@ -3115,41 +2966,19 @@ class StubGenerator: public StubCodeGenerator {
     __ bind            (L_do44);
 
     // load the 11th round key to vKey1
-    __ li              (keypos, 176);
-    __ lvx             (vKey1, keypos, key);
-    __ li              (keypos, 160);
-    __ lvx             (vTmp1, keypos, key);
-    __ vec_perm        (vKey1, vTmp1, vKey1, keyPerm);
+    __ load_word_vector_unaligned(vKey1, 160, key, tmp);
 
     // 1st round
     __ vxor            (vRet, vRet, vKey1);
 
     __ bind            (L_doLast);
 
-    // load the 10th round key to vKey1
-    __ li              (keypos, 144);
-    __ lvx             (vKey2, keypos, key);
-    __ vec_perm        (vKey1, vKey2, vTmp1, keyPerm);
-
-    // load the 9th round key to vKey2
-    __ li              (keypos, 128);
-    __ lvx             (vKey3, keypos, key);
-    __ vec_perm        (vKey2, vKey3, vKey2, keyPerm);
-
-    // load the 8th round key to vKey3
-    __ li              (keypos, 112);
-    __ lvx             (vKey4, keypos, key);
-    __ vec_perm        (vKey3, vKey4, vKey3, keyPerm);
-
-    // load the 7th round key to vKey4
-    __ li              (keypos, 96);
-    __ lvx             (vKey5, keypos, key);
-    __ vec_perm        (vKey4, vKey5, vKey4, keyPerm);
-
-    // load the 6th round key to vKey5
-    __ li              (keypos, 80);
-    __ lvx             (vTmp1, keypos, key);
-    __ vec_perm        (vKey5, vTmp1, vKey5, keyPerm);
+    // load the 10th - 6th round key to vKey1 - vKey5
+    __ load_word_vector_unaligned(vKey1, 144, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 128, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 112, key, tmp);
+    __ load_word_vector_unaligned(vKey4, 96, key, tmp);
+    __ load_word_vector_unaligned(vKey5, 80, key, tmp);
 
     // last 10th - 6th rounds
     __ vncipher        (vRet, vRet, vKey1);
@@ -3158,29 +2987,12 @@ class StubGenerator: public StubCodeGenerator {
     __ vncipher        (vRet, vRet, vKey4);
     __ vncipher        (vRet, vRet, vKey5);
 
-    // load the 5th round key to vKey1
-    __ li              (keypos, 64);
-    __ lvx             (vKey2, keypos, key);
-    __ vec_perm        (vKey1, vKey2, vTmp1, keyPerm);
-
-    // load the 4th round key to vKey2
-    __ li              (keypos, 48);
-    __ lvx             (vKey3, keypos, key);
-    __ vec_perm        (vKey2, vKey3, vKey2, keyPerm);
-
-    // load the 3rd round key to vKey3
-    __ li              (keypos, 32);
-    __ lvx             (vKey4, keypos, key);
-    __ vec_perm        (vKey3, vKey4, vKey3, keyPerm);
-
-    // load the 2nd round key to vKey4
-    __ li              (keypos, 16);
-    __ lvx             (vKey5, keypos, key);
-    __ vec_perm        (vKey4, vKey5, vKey4, keyPerm);
-
-    // load the 1st round key to vKey5
-    __ lvx             (vTmp1, key);
-    __ vec_perm        (vKey5, vTmp1, vKey5, keyPerm);
+    // load the 5th - 1st round key to vKey1 - vKey5
+    __ load_word_vector_unaligned(vKey1, 64, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 48, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 32, key, tmp);
+    __ load_word_vector_unaligned(vKey4, 16, key, tmp);
+    __ load_word_vector_unaligned(vKey5, 0, key, tmp);
 
     // last 5th - 1th rounds
     __ vncipher        (vRet, vRet, vKey1);
@@ -3189,23 +3001,8 @@ class StubGenerator: public StubCodeGenerator {
     __ vncipher        (vRet, vRet, vKey4);
     __ vncipherlast    (vRet, vRet, vKey5);
 
-#ifdef VM_LITTLE_ENDIAN
-    // toPerm = 0x0F0E0D0C0B0A09080706050403020100
-    __ lvsl            (toPerm, keypos); // keypos is a multiple of 16
-    __ vxor            (toPerm, toPerm, fSplt);
-
-    // Swap Bytes
-    __ vperm           (vRet, vRet, vRet, toPerm);
-#endif
-
     // store result (unaligned)
-    // Note: We can't use a read-modify-write sequence which touches additional Bytes.
-    Register lo = temp, hi = fifteen; // Reuse
-    __ vsldoi          (vTmp1, vRet, vRet, 8);
-    __ mfvrd           (hi, vRet);
-    __ mfvrd           (lo, vTmp1);
-    __ std             (hi, 0 LITTLE_ENDIAN_ONLY(+ 8), to);
-    __ std             (lo, 0 BIG_ENDIAN_ONLY(+ 8), to);
+    __ store_byte_vector_unaligned(vRet, 0, to, tmp, vp);
 
     __ blr();
 
@@ -3216,6 +3013,306 @@ class StubGenerator: public StubCodeGenerator {
      return start;
   }
 
+  // ==========================================================================
+  // AES helper functions for PPC64
+  //
+  // These emit the AES round instructions.
+  // Each call to these helpers emits a sequence of vcipher/vncipher
+  // instructions.
+  //
+  // ==========================================================================
+  // Emits the AES encrypt round instructions.
+  //
+  // vRet:    in/out — the AES state (plaintext in, ciphertext out)
+  // key:     register holding pointer to expanded key array
+  // keylen:  register holding key length (44/52/60)
+  //
+  void aes_encrypt_rounds(VectorRegister vRet,
+                          Register key, Register keylen, Register tmp,
+                          VectorRegister vKey1, VectorRegister vKey2,
+                          VectorRegister vKey3, VectorRegister vKey4) {
+    Label L_doLast;
+
+    // round 0: AddRoundKey
+    __ load_word_vector_unaligned(vKey1, 0, key, tmp);
+    __ vxor            (vRet, vRet, vKey1);
+
+    // rounds 2-5
+    __ load_word_vector_unaligned(vKey1, 16, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 32, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 48, key, tmp);
+    __ load_word_vector_unaligned(vKey4, 64, key, tmp);
+    __ vcipher         (vRet, vRet, vKey1);
+    __ vcipher         (vRet, vRet, vKey2);
+    __ vcipher         (vRet, vRet, vKey3);
+    __ vcipher         (vRet, vRet, vKey4);
+
+    // rounds 6-9
+    __ load_word_vector_unaligned(vKey1, 80, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 96, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 112, key, tmp);
+    __ load_word_vector_unaligned(vKey4, 128, key, tmp);
+    __ vcipher         (vRet, vRet, vKey1);
+    __ vcipher         (vRet, vRet, vKey2);
+    __ vcipher         (vRet, vRet, vKey3);
+    __ vcipher         (vRet, vRet, vKey4);
+
+    // rounds 10-11
+    __ load_word_vector_unaligned(vKey1, 144, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 160, key, tmp);
+
+    __ cmpwi           (CR0, keylen, 44);   // AES-128 -> final rounds
+    __ beq             (CR0, L_doLast);
+
+    __ vcipher         (vRet, vRet, vKey1);
+    __ vcipher         (vRet, vRet, vKey2);
+
+    // rounds 12-13
+    __ load_word_vector_unaligned(vKey1, 176, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 192, key, tmp);
+
+    __ cmpwi           (CR0, keylen, 52);   // AES-192 -> final rounds
+    __ beq             (CR0, L_doLast);
+#ifdef ASSERT
+      __ cmpwi         (CR0, keylen, 60);
+      __ asm_assert_eq(FILE_AND_LINE ": aes_encrypt_rounds - invalid key length");
+#endif
+
+    __ vcipher         (vRet, vRet, vKey1);
+    __ vcipher         (vRet, vRet, vKey2);
+
+    // rounds 14-15
+    __ load_word_vector_unaligned(vKey1, 208, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 224, key, tmp);
+
+    __ bind(L_doLast);
+    __ vcipher         (vRet, vRet, vKey1);
+    __ vcipherlast     (vRet, vRet, vKey2);
+  }
+
+
+  // ==========================================================================
+  // Emits the AES decrypt round instructions.
+  //
+  // vRet:    in/out — the AES state (ciphertext in, plaintext out)
+  // key:     register holding pointer to expanded key array
+  // keylen:  register holding key length (44/52/60)
+  //
+  void aes_decrypt_rounds(VectorRegister vRet,
+                          Register key, Register keylen, Register tmp,
+                          VectorRegister vKey1, VectorRegister vKey2,
+                          VectorRegister vKey3, VectorRegister vKey4,
+                          VectorRegister vKey5) {
+    Label L_doLast, L_do44, L_do52;
+
+    __ cmpwi           (CR0, keylen, 44);
+    __ beq             (CR0, L_do44);
+
+    __ cmpwi           (CR0, keylen, 52);
+    __ beq             (CR0, L_do52);
+
+#ifdef ASSERT
+      __ cmpwi         (CR0, keylen, 60);
+      __ asm_assert_eq(FILE_AND_LINE ": aes_decrypt_rounds - invalid key length");
+#endif
+    // ---- AES-256: round keys 15-11 ----
+    __ load_word_vector_unaligned(vKey1, 224, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 208, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 192, key, tmp);
+    __ load_word_vector_unaligned(vKey4, 176, key, tmp);
+    __ load_word_vector_unaligned(vKey5, 160, key, tmp);
+
+    __ vxor            (vRet, vRet, vKey1);
+    __ vncipher        (vRet, vRet, vKey2);
+    __ vncipher        (vRet, vRet, vKey3);
+    __ vncipher        (vRet, vRet, vKey4);
+    __ vncipher        (vRet, vRet, vKey5);
+    __ b               (L_doLast);
+
+    __ align(32);
+    // ---- AES-192: round keys 13-11 ----
+    __ bind            (L_do52);
+    __ load_word_vector_unaligned(vKey1, 192, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 176, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 160, key, tmp);
+
+    __ vxor            (vRet, vRet, vKey1);
+    __ vncipher        (vRet, vRet, vKey2);
+    __ vncipher        (vRet, vRet, vKey3);
+    __ b               (L_doLast);
+
+    __ align(32);
+    // ---- AES-128: round key 11 ----
+    __ bind            (L_do44);
+    __ load_word_vector_unaligned(vKey1, 160, key, tmp);
+    __ vxor            (vRet, vRet, vKey1);
+
+    // ---- Common rounds 10-1 ----
+    __ bind            (L_doLast);
+    __ load_word_vector_unaligned(vKey1, 144, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 128, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 112, key, tmp);
+    __ load_word_vector_unaligned(vKey4, 96,  key, tmp);
+    __ load_word_vector_unaligned(vKey5, 80,  key, tmp);
+
+    __ vncipher        (vRet, vRet, vKey1);
+    __ vncipher        (vRet, vRet, vKey2);
+    __ vncipher        (vRet, vRet, vKey3);
+    __ vncipher        (vRet, vRet, vKey4);
+    __ vncipher        (vRet, vRet, vKey5);
+    __ load_word_vector_unaligned(vKey1, 64, key, tmp);
+    __ load_word_vector_unaligned(vKey2, 48, key, tmp);
+    __ load_word_vector_unaligned(vKey3, 32, key, tmp);
+    __ load_word_vector_unaligned(vKey4, 16, key, tmp);
+    __ load_word_vector_unaligned(vKey5, 0,  key, tmp);
+    __ vncipher        (vRet, vRet, vKey1);
+    __ vncipher        (vRet, vRet, vKey2);
+    __ vncipher        (vRet, vRet, vKey3);
+    __ vncipher        (vRet, vRet, vKey4);
+    __ vncipherlast    (vRet, vRet, vKey5);
+  }
+
+  // ==========================================================================
+  // CBC Encrypt stub — using helper functions
+  //   from:      R3_ARG1          - source byte array address (plaintext)
+  //   to:        R4_ARG2          - destination byte array address (ciphertext)
+  //   key:       R5_ARG3          - round key array
+  //   rvec:      R6_ARG4          - r vector byte array address (initialization vector)
+  //   input_len: R7_ARG5          - length of input in bytes
+  //
+  // Returns:
+  //   R3_RET - number of bytes processed
+  //
+  address generate_cipherBlockChaining_encryptAESCrypt() {
+    assert(UseAESIntrinsics, "need AES instructions support");
+    StubId stub_id = StubId::stubgen_cipherBlockChaining_encryptAESCrypt_id;
+    StubCodeMark mark(this, stub_id);
+
+    address start = __ function_entry();
+
+    Label L_enc_loop;
+
+    Register from       = R3_ARG1;
+    Register to         = R4_ARG2;
+    Register key        = R5_ARG3;
+    Register rvec       = R6_ARG4;
+    Register input_len  = R7_ARG5;
+
+    Register keylen     = R8;
+    Register tmp        = R9;
+    Register len        = R10;
+
+    VectorRegister vRet  = VR0;
+    VectorRegister vKey1 = VR1;
+    VectorRegister vKey2 = VR2;
+    VectorRegister vKey3 = VR3;
+    VectorRegister vKey4 = VR4;
+    VectorRegister vIn   = VR5;
+    VectorRegister vp    = VR6;   // permute vector for P8 LE byte accesses
+    VectorRegister vTmp  = VR7;
+
+    __ mr              (len, input_len);
+
+    // vp must be computed once, before any byte vector access. Clobbers R0.
+    __ compute_vp_for_byte_vector_unaligned(vp, /*temp*/ vRet);
+
+    __ load_byte_vector_unaligned(vRet, 0, rvec, tmp, vp);
+
+    __ lwz             (keylen, arrayOopDesc::length_offset_in_bytes() -
+                                arrayOopDesc::base_offset_in_bytes(T_INT), key);
+
+    __ align(32);
+    __ bind(L_enc_loop);
+    __ load_byte_vector_unaligned(vIn, 0, from, tmp, vp);
+    __ addi            (from, from, 16);
+    __ vxor            (vRet, vRet, vIn);                      // CBC XOR
+    aes_encrypt_rounds(vRet, key, keylen, tmp, vKey1, vKey2, vKey3, vKey4);
+    __ store_byte_vector_unaligned(vRet, 0, to, tmp, vp, vTmp);
+    __ addi            (to, to, 16);
+    __ addic_          (len, len, -16);
+    __ bne             (CR0, L_enc_loop);
+
+    // save the last ciphertext block in rvec; it is the IV for the next call
+    __ store_byte_vector_unaligned(vRet, 0, rvec, tmp, vp, vTmp);
+    __ mr              (R3_RET, input_len);
+    __ blr();
+
+    return start;
+  }
+
+  // ==========================================================================
+  //  CBC Decrypt stub
+  //  Arguments:
+  //    R3_ARG1 - from:      source byte array address (ciphertext)
+  //    R4_ARG2 - to:        destination byte array address (plaintext)
+  //    R5_ARG3 - key:       round key array
+  //    R6_ARG4 - rvec:      r vector byte array address (in/out), holds the
+  //                         initialization vector on entry and is updated with
+  //                         the last ciphertext block on exit
+  //    R7_ARG5 - input_len: length of input in bytes, a multiple of 16
+  //
+  //  Returns:
+  //    R3_RET  - number of bytes processed
+  // ==========================================================================
+
+  address generate_cipherBlockChaining_decryptAESCrypt() {
+    assert(UseAESIntrinsics, "need AES instructions support");
+    StubId stub_id = StubId::stubgen_cipherBlockChaining_decryptAESCrypt_id;
+    StubCodeMark mark(this, stub_id);
+
+    address start = __ function_entry();
+
+    Label L_dec_loop;
+
+    Register from       = R3_ARG1;
+    Register to         = R4_ARG2;
+    Register key        = R5_ARG3;
+    Register rvec       = R6_ARG4;
+    Register input_len  = R7_ARG5;
+
+    Register keylen     = R8;
+    Register tmp        = R9;
+    Register len        = R10;
+
+    VectorRegister vRet     = VR0;
+    VectorRegister vKey1    = VR1;
+    VectorRegister vKey2    = VR2;
+    VectorRegister vKey3    = VR3;
+    VectorRegister vKey4    = VR4;
+    VectorRegister vKey5    = VR5;
+    VectorRegister vIV      = VR6;
+    VectorRegister vSavedCT = VR7;
+    VectorRegister vp       = VR8;   // permute vector for P8 LE byte accesses
+    VectorRegister vTmp     = VR9;
+    __ mr              (len, input_len);
+    // vp must be computed before any byte vector access. Clobbers R0.
+    __ compute_vp_for_byte_vector_unaligned(vp, /*temp*/ vRet);
+
+    __ load_byte_vector_unaligned(vIV, 0, rvec, tmp, vp);
+
+    __ lwz             (keylen, arrayOopDesc::length_offset_in_bytes() -
+                                arrayOopDesc::base_offset_in_bytes(T_INT), key);
+
+    __ align(32);
+    __ bind(L_dec_loop);
+    __ load_byte_vector_unaligned(vRet, 0, from, tmp, vp);
+    __ addi            (from, from, 16);
+    __ vor             (vSavedCT, vRet, vRet);      // AES will destroy vRet
+    aes_decrypt_rounds(vRet, key, keylen, tmp, vKey1, vKey2, vKey3, vKey4, vKey5);
+    __ vxor            (vRet, vRet, vIV);           // CBC XOR (after decrypt)
+    __ vor             (vIV, vSavedCT, vSavedCT);   // IV = previous ciphertext
+    __ store_byte_vector_unaligned(vRet, 0, to, tmp, vp, vTmp);
+    __ addi            (to, to, 16);
+    __ addic_          (len, len, -16);
+    __ bne             (CR0, L_dec_loop);
+
+    __ store_byte_vector_unaligned(vIV, 0, rvec, tmp, vp, vTmp);
+    __ mr              (R3_RET, input_len);
+    __ blr();
+
+    return start;
+  }
+
   address generate_sha256_implCompress(StubId stub_id) {
     assert(UseSHA, "need SHA instructions");
     bool multi_block;
@@ -5092,6 +5189,8 @@ void generate_lookup_secondary_supers_table_stub() {
     if (UseAESIntrinsics) {
       StubRoutines::_aescrypt_encryptBlock = generate_aescrypt_encryptBlock();
       StubRoutines::_aescrypt_decryptBlock = generate_aescrypt_decryptBlock();
+      StubRoutines::_cipherBlockChaining_encryptAESCrypt = generate_cipherBlockChaining_encryptAESCrypt();
+      StubRoutines::_cipherBlockChaining_decryptAESCrypt = generate_cipherBlockChaining_decryptAESCrypt();
     }
 
     if (UseSHA256Intrinsics) {

src/hotspot/cpu/ppc/templateInterpreterGenerator_ppc.cpp

@@ -1162,7 +1162,7 @@ address TemplateInterpreterGenerator::generate_math_entry(AbstractInterpreter::M
   __ resize_frame_absolute(R21_sender_SP, R11_scratch1, R0);
   __ blr();
 
-  __ flush();
+  __ invalidate_icache();
 
   return entry;
 }
@@ -1179,7 +1179,7 @@ address TemplateInterpreterGenerator::generate_Float_floatToFloat16_entry() {
   __ resize_frame_absolute(R21_sender_SP, R11_scratch1, R0);
   __ blr();
 
-  __ flush();
+  __ invalidate_icache();
 
   return entry;
 }
@@ -1200,7 +1200,7 @@ address TemplateInterpreterGenerator::generate_Float_float16ToFloat_entry() {
   __ resize_frame_absolute(R21_sender_SP, R11_scratch1, R0);
   __ blr();
 
-  __ flush();
+  __ invalidate_icache();
 
   return entry;
 }
@@ -1484,21 +1484,10 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) {
   // In order for GC to work, don't clear the last_Java_sp until after
   // blocking.
 
-  //=============================================================================
-  // Switch thread to "native transition" state before reading the
-  // synchronization state. This additional state is necessary
-  // because reading and testing the synchronization state is not
-  // atomic w.r.t. GC, as this scenario demonstrates: Java thread A,
-  // in _thread_in_native state, loads _not_synchronized and is
-  // preempted. VM thread changes sync state to synchronizing and
-  // suspends threads for GC. Thread A is resumed to finish this
-  // native method, but doesn't block here since it didn't see any
-  // synchronization in progress, and escapes.
-
   // We use release_store_fence to update values like the thread state, where
   // we don't want the current thread to continue until all our prior memory
   // accesses (including the new thread state) are visible to other threads.
-  __ li(R0/*thread_state*/, _thread_in_native_trans);
+  __ li(R0/*thread_state*/, _thread_in_vm);
   __ release();
   __ stw(R0/*thread_state*/, thread_(thread_state));
   if (!UseSystemMemoryBarrier) {
@@ -1506,9 +1495,8 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) {
   }
 
   // Now before we return to java we must look for a current safepoint
-  // (a new safepoint can not start since we entered native_trans).
-  // We must check here because a current safepoint could be modifying
-  // the callers registers right this moment.
+  // (a new safepoint can not start since we entered _thread_in_vm).
+  // We must check here because a current safepoint could be in progress.
 
   // Acquire isn't strictly necessary here because of the fence, but
   // sync_state is declared to be volatile, so we do it anyway
@@ -1538,7 +1526,7 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) {
   //=============================================================================
   // <<<<<< Back in Interpreter Frame >>>>>
 
-  // We are in thread_in_native_trans here and back in the normal
+  // We are in _thread_in_vm here and back in the normal
   // interpreter frame. We don't have to do anything special about
   // safepoints and we can switch to Java mode anytime we are ready.
 

src/hotspot/cpu/ppc/templateTable_ppc_64.cpp

@@ -2609,7 +2609,8 @@ void TemplateTable::jvmti_post_field_access(Register Rcache, Register Rscratch,
       // Restore object pointer.
       __ pop_ptr(R17_tos);
       __ verify_oop(R17_tos);
-    } else {
+    }
+    if (Rcache.is_volatile()) {
       // Cache is still needed to get class or obj.
       __ load_field_entry(Rcache, Rscratch);
     }
@@ -2646,10 +2647,12 @@ void TemplateTable::getfield_or_static(int byte_no, bool is_static, RewriteContr
                  Rscratch      = R11_scratch1; // used by load_field_cp_cache_entry
                  // R12_scratch2 used by load_field_cp_cache_entry
 
-  static address field_branch_table[number_of_states],
+  static address field_rw_branch_table[number_of_states],
+                 field_norw_branch_table[number_of_states],
                  static_branch_table[number_of_states];
 
-  address* branch_table = (is_static || rc == may_not_rewrite) ? static_branch_table : field_branch_table;
+  address* branch_table = is_static ? static_branch_table :
+    (rc == may_rewrite ? field_rw_branch_table : field_norw_branch_table);
 
   // Get field offset.
   resolve_cache_and_index_for_field(byte_no, Rcache, Rscratch);
@@ -2697,14 +2700,7 @@ void TemplateTable::getfield_or_static(int byte_no, bool is_static, RewriteContr
 #ifdef ASSERT
   __ bind(LFlagInvalid);
   __ stop("got invalid flag");
-#endif
 
-  if (!is_static && rc == may_not_rewrite) {
-    // We reuse the code from is_static.  It's jumped to via the table above.
-    return;
-  }
-
-#ifdef ASSERT
   // __ bind(Lvtos);
   address pc_before_fence = __ pc();
   __ fence(); // Volatile entry point (one instruction before non-volatile_entry point).
@@ -3322,7 +3318,7 @@ void TemplateTable::fast_storefield(TosState state) {
     {
       Label is_flat, done;
       __ test_field_is_flat(Rflags, is_flat);
-      __ null_check_throw(Rclass_or_obj, -1, Rscratch);
+      __ null_check_throw(R17_tos, -1, Rscratch);
       do_oop_store(_masm, Rclass_or_obj, Roffset, R17_tos, Rscratch, Rscratch2, Rscratch3, IN_HEAP);
       __ b(done);
       __ bind(is_flat);
@@ -3384,7 +3380,7 @@ void TemplateTable::fast_accessfield(TosState state) {
   Label LisVolatile;
   ByteSize cp_base_offset = ConstantPoolCache::base_offset();
 
-  const Register Rcache        = R3_ARG1,
+  const Register Rcache        = R31, // Needs to survive C call.
                  Rclass_or_obj = R17_tos,
                  Roffset       = R22_tmp2,
                  Rflags        = R23_tmp3,

src/hotspot/cpu/ppc/upcallLinker_ppc.cpp

@@ -1,6 +1,6 @@
 /*
- * Copyright (c) 2020, 2025, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2023, 2025 SAP SE. All rights reserved.
+ * Copyright (c) 2020, 2026, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2023, 2026 SAP SE. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * This code is free software; you can redistribute it and/or modify it
@@ -243,7 +243,7 @@ address UpcallLinker::make_upcall_stub(jobject receiver, Symbol* signature,
 
   //////////////////////////////////////////////////////////////////////////////
 
-  _masm->flush();
+  // Code will be copied. No ICache sync required.
 
 #ifndef PRODUCT
   stringStream ss;

src/hotspot/cpu/ppc/vm_version_ppc.cpp

@@ -514,7 +514,7 @@ void VM_Version::determine_features() {
   a->blr();
 
   uint32_t *code_end = (uint32_t *)a->pc();
-  a->flush();
+  a->invalidate_icache();
   _features = VM_Version::unknown_m;
 
   // Print the detection code.
@@ -570,7 +570,7 @@ void VM_Version::config_dscr() {
   a->blr();
 
   uint32_t *code_end = (uint32_t *)a->pc();
-  a->flush();
+  a->invalidate_icache();
 
   // Print the detection code.
   if (PrintAssembly) {

src/hotspot/cpu/ppc/vtableStubs_ppc_64.cpp

@@ -1,6 +1,6 @@
 /*
  * Copyright (c) 1997, 2026, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2012, 2025 SAP SE. All rights reserved.
+ * Copyright (c) 2012, 2026 SAP SE. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * This code is free software; you can redistribute it and/or modify it
@@ -124,7 +124,7 @@ VtableStub* VtableStubs::create_vtable_stub(int vtable_index, bool caller_is_c1)
   __ mtctr(R12_scratch2);
   __ bctr();
 
-  masm->flush();
+  masm->invalidate_icache();
   bookkeeping(masm, tty, s, npe_addr, ame_addr, true, vtable_index, slop_bytes, 0);
 
   return s;
@@ -224,7 +224,7 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index, bool caller_is_c1)
   __ mtctr(R11_scratch1);
   __ bctr();
 
-  masm->flush();
+  masm->invalidate_icache();
   bookkeeping(masm, tty, s, npe_addr, ame_addr, false, itable_index, slop_bytes, 0);
 
   return s;

src/hotspot/cpu/riscv/assembler_riscv.hpp

@@ -514,20 +514,138 @@ protected:
     rdy = 0b111,     // in instruction's rm field, selects dynamic rounding mode.In Rounding Mode register, Invalid.
   };
 
+  // Efficient reading and writing of unaligned data in platform-specific byte ordering
+  // RISC-V needs to check for alignment.
+
+  static inline u2 get_native_u2(address p) {
+    if ((intptr_t(p) & 1) == 0) {
+      return *(u2*)p;
+    } else {
+      return ((u2)(p[1]) << 8) |
+             ((u2)(p[0]));
+    }
+  }
+
+  static inline u4 get_native_u4(address p) {
+    switch (intptr_t(p) & 3) {
+      case 0:
+        return *(u4*)p;
+
+      case 2:
+        return ((u4)(((u2*)p)[1]) << 16) |
+               ((u4)(((u2*)p)[0]));
+
+      default:
+        return ((u4)(p[3]) << 24) |
+               ((u4)(p[2]) << 16) |
+               ((u4)(p[1]) <<  8) |
+               ((u4)(p[0]));
+    }
+  }
+
+  static inline u8 get_native_u8(address p) {
+    switch (intptr_t(p) & 7) {
+      case 0:
+        return *(u8*)p;
+
+      case 4:
+        return ((u8)(((u4*)p)[1]) << 32) |
+               ((u8)(((u4*)p)[0]));
+
+      case 2:
+      case 6:
+        return ((u8)(((u2*)p)[3]) << 48) |
+               ((u8)(((u2*)p)[2]) << 32) |
+               ((u8)(((u2*)p)[1]) << 16) |
+               ((u8)(((u2*)p)[0]));
+
+      default:
+        return ((u8)(p[7]) << 56) |
+               ((u8)(p[6]) << 48) |
+               ((u8)(p[5]) << 40) |
+               ((u8)(p[4]) << 32) |
+               ((u8)(p[3]) << 24) |
+               ((u8)(p[2]) << 16) |
+               ((u8)(p[1]) <<  8) |
+               ((u8)(p[0]));
+    }
+  }
+
+  static inline void put_native_u2(address p, u2 x) {
+    if ((intptr_t(p) & 1) == 0) {
+      *(u2*)p = x;
+    } else {
+      p[1] = x >> 8;
+      p[0] = x;
+    }
+  }
+
+  static inline void put_native_u4(address p, u4 x) {
+    switch (intptr_t(p) & 3) {
+      case 0:
+        *(u4*)p = x;
+        break;
+
+      case 2:
+        ((u2*)p)[1] = x >> 16;
+        ((u2*)p)[0] = x;
+        break;
+
+      default:
+        ((u1*)p)[3] = x >> 24;
+        ((u1*)p)[2] = x >> 16;
+        ((u1*)p)[1] = x >>  8;
+        ((u1*)p)[0] = x;
+        break;
+    }
+  }
+
+  static inline void put_native_u8(address p, u8 x) {
+    switch (intptr_t(p) & 7) {
+      case 0:
+        *(u8*)p = x;
+        break;
+
+      case 4:
+        ((u4*)p)[1] = x >> 32;
+        ((u4*)p)[0] = x;
+        break;
+
+      case 2:
+      case 6:
+        ((u2*)p)[3] = x >> 48;
+        ((u2*)p)[2] = x >> 32;
+        ((u2*)p)[1] = x >> 16;
+        ((u2*)p)[0] = x;
+        break;
+
+      default:
+        ((u1*)p)[7] = x >> 56;
+        ((u1*)p)[6] = x >> 48;
+        ((u1*)p)[5] = x >> 40;
+        ((u1*)p)[4] = x >> 32;
+        ((u1*)p)[3] = x >> 24;
+        ((u1*)p)[2] = x >> 16;
+        ((u1*)p)[1] = x >>  8;
+        ((u1*)p)[0] = x;
+        break;
+    }
+  }
+
   // handle unaligned access
   static inline uint16_t ld_c_instr(address addr) {
-    return Bytes::get_native_u2(addr);
+    return get_native_u2(addr);
   }
   static inline void sd_c_instr(address addr, uint16_t c_instr) {
-    Bytes::put_native_u2(addr, c_instr);
+    put_native_u2(addr, c_instr);
   }
 
   // handle unaligned access
   static inline uint32_t ld_instr(address addr) {
-    return Bytes::get_native_u4(addr);
+    return get_native_u4(addr);
   }
   static inline void sd_instr(address addr, uint32_t instr) {
-    Bytes::put_native_u4(addr, instr);
+    put_native_u4(addr, instr);
   }
 
   static inline uint32_t extract(uint32_t val, unsigned msb, unsigned lsb) {

src/hotspot/cpu/riscv/bytes_riscv.hpp

@@ -1,167 +0,0 @@
-/*
- * Copyright (c) 1997, 2019, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2012, 2016 SAP SE. All rights reserved.
- * Copyright (c) 2020, 2022, Huawei Technologies Co., Ltd. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#ifndef CPU_RISCV_BYTES_RISCV_HPP
-#define CPU_RISCV_BYTES_RISCV_HPP
-
-#include "memory/allStatic.hpp"
-#include "utilities/byteswap.hpp"
-
-class Bytes: AllStatic {
- public:
-  // Efficient reading and writing of unaligned unsigned data in platform-specific byte ordering
-  // RISCV needs to check for alignment.
-
-  static inline u2 get_native_u2(address p) {
-    if ((intptr_t(p) & 1) == 0) {
-      return *(u2*)p;
-    } else {
-      return ((u2)(p[1]) << 8) |
-             ((u2)(p[0]));
-    }
-  }
-
-  static inline u4 get_native_u4(address p) {
-    switch (intptr_t(p) & 3) {
-      case 0:
-        return *(u4*)p;
-
-      case 2:
-        return ((u4)(((u2*)p)[1]) << 16) |
-               ((u4)(((u2*)p)[0]));
-
-      default:
-        return ((u4)(p[3]) << 24) |
-               ((u4)(p[2]) << 16) |
-               ((u4)(p[1]) <<  8) |
-               ((u4)(p[0]));
-    }
-  }
-
-  static inline u8 get_native_u8(address p) {
-    switch (intptr_t(p) & 7) {
-      case 0:
-        return *(u8*)p;
-
-      case 4:
-        return ((u8)(((u4*)p)[1]) << 32) |
-               ((u8)(((u4*)p)[0]));
-
-      case 2:
-      case 6:
-        return ((u8)(((u2*)p)[3]) << 48) |
-               ((u8)(((u2*)p)[2]) << 32) |
-               ((u8)(((u2*)p)[1]) << 16) |
-               ((u8)(((u2*)p)[0]));
-
-      default:
-        return ((u8)(p[7]) << 56) |
-               ((u8)(p[6]) << 48) |
-               ((u8)(p[5]) << 40) |
-               ((u8)(p[4]) << 32) |
-               ((u8)(p[3]) << 24) |
-               ((u8)(p[2]) << 16) |
-               ((u8)(p[1]) <<  8) |
-               ((u8)(p[0]));
-    }
-  }
-
-  static inline void put_native_u2(address p, u2 x) {
-    if ((intptr_t(p) & 1) == 0) {
-      *(u2*)p = x;
-    } else {
-      p[1] = x >> 8;
-      p[0] = x;
-    }
-  }
-
-  static inline void put_native_u4(address p, u4 x) {
-    switch (intptr_t(p) & 3) {
-      case 0:
-        *(u4*)p = x;
-        break;
-
-      case 2:
-        ((u2*)p)[1] = x >> 16;
-        ((u2*)p)[0] = x;
-        break;
-
-      default:
-        ((u1*)p)[3] = x >> 24;
-        ((u1*)p)[2] = x >> 16;
-        ((u1*)p)[1] = x >>  8;
-        ((u1*)p)[0] = x;
-        break;
-    }
-  }
-
-  static inline void put_native_u8(address p, u8 x) {
-    switch (intptr_t(p) & 7) {
-      case 0:
-        *(u8*)p = x;
-        break;
-
-      case 4:
-        ((u4*)p)[1] = x >> 32;
-        ((u4*)p)[0] = x;
-        break;
-
-      case 2:
-      case 6:
-        ((u2*)p)[3] = x >> 48;
-        ((u2*)p)[2] = x >> 32;
-        ((u2*)p)[1] = x >> 16;
-        ((u2*)p)[0] = x;
-        break;
-
-      default:
-        ((u1*)p)[7] = x >> 56;
-        ((u1*)p)[6] = x >> 48;
-        ((u1*)p)[5] = x >> 40;
-        ((u1*)p)[4] = x >> 32;
-        ((u1*)p)[3] = x >> 24;
-        ((u1*)p)[2] = x >> 16;
-        ((u1*)p)[1] = x >>  8;
-        ((u1*)p)[0] = x;
-        break;
-    }
-  }
-
-#ifndef VM_LITTLE_ENDIAN
-#error RISC-V is little endian, the preprocessor macro VM_LITTLE_ENDIAN should be defined.
-#endif
-
-  // Efficient reading and writing of unaligned unsigned data in Java byte ordering (i.e. big-endian ordering)
-  static inline u2 get_Java_u2(address p) { return byteswap(get_native_u2(p)); }
-  static inline u4 get_Java_u4(address p) { return byteswap(get_native_u4(p)); }
-  static inline u8 get_Java_u8(address p) { return byteswap(get_native_u8(p)); }
-
-  static inline void put_Java_u2(address p, u2 x) { put_native_u2(p, byteswap(x)); }
-  static inline void put_Java_u4(address p, u4 x) { put_native_u4(p, byteswap(x)); }
-  static inline void put_Java_u8(address p, u8 x) { put_native_u8(p, byteswap(x)); }
-};
-
-#endif // CPU_RISCV_BYTES_RISCV_HPP

src/hotspot/cpu/riscv/c1_LIRAssembler_riscv.cpp

@@ -36,6 +36,7 @@
 #include "ci/ciInlineKlass.hpp"
 #include "ci/ciInstance.hpp"
 #include "ci/ciObjArrayKlass.hpp"
+#include "code/aotCodeCache.hpp"
 #include "code/compiledIC.hpp"
 #include "gc/shared/collectedHeap.hpp"
 #include "nativeInst_riscv.hpp"
@@ -43,6 +44,7 @@
 #include "oops/oop.inline.hpp"
 #include "runtime/frame.inline.hpp"
 #include "runtime/sharedRuntime.hpp"
+#include "runtime/threadIdentifier.hpp"
 #include "utilities/powerOfTwo.hpp"
 #include "vmreg_riscv.inline.hpp"
 
@@ -441,6 +443,19 @@ void LIR_Assembler::const2reg(LIR_Opr src, LIR_Opr dest, LIR_PatchCode patch_cod
 
     case T_LONG:
       assert(patch_code == lir_patch_none, "no patching handled here");
+#if INCLUDE_CDS
+      if (AOTCodeCache::is_on_for_dump()) {
+        address b = c->as_pointer();
+        if (b == (address)ThreadIdentifier::unsafe_offset()) {
+          __ la(dest->as_register_lo(), ExternalAddress(b));
+          break;
+        }
+        if (AOTRuntimeConstants::contains(b)) {
+          __ load_aotrc_address(dest->as_register_lo(), b);
+          break;
+        }
+      }
+#endif
       __ mv(dest->as_register_lo(), (intptr_t)c->as_jlong());
       break;
 

src/hotspot/cpu/riscv/c1_LIRAssembler_riscv.hpp

@@ -69,7 +69,7 @@ private:
     _call_stub_size = 11 * MacroAssembler::instruction_size +
                       1 * MacroAssembler::instruction_size + wordSize,
     // See emit_exception_handler for detail
-    _exception_handler_size = DEBUG_ONLY(256) NOT_DEBUG(32), // or smaller
+    _exception_handler_size = DEBUG_ONLY(1*K) NOT_DEBUG(175), // or smaller
     // See emit_deopt_handler for detail
     // far_call (2) + j (1)
     _deopt_handler_size = 1 * MacroAssembler::instruction_size +

src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.cpp

@@ -84,10 +84,8 @@ void C2_MacroAssembler::fast_lock(Register obj, Register box,
   // Finish fast lock unsuccessfully. slow_path MUST branch to with flag != 0
   Label slow_path;
 
-  if (UseObjectMonitorTable) {
-    // Clear cache in case fast locking succeeds or we need to take the slow-path.
-    sd(zr, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
-  }
+  // Clear cache in case fast locking succeeds or we need to take the slow-path.
+  sd(zr, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
 
   if (DiagnoseSyncOnValueBasedClasses != 0) {
     load_klass(tmp1, obj);
@@ -149,61 +147,56 @@ void C2_MacroAssembler::fast_lock(Register obj, Register box,
     const ByteSize omc_monitor_offset = OMCache::monitor_offset();
     const ByteSize omc_obj_offset     = OMCache::obj_offset();
 
-    if (!UseObjectMonitorTable) {
-      assert(tmp1_monitor == tmp1_mark, "should be the same here");
-    } else {
-      const Register tmp2_hash = tmp2;
-      const Register tmp3_bucket = tmp3;
-      Label monitor_found;
+    const Register tmp2_hash = tmp2;
+    const Register tmp3_bucket = tmp3;
+    Label monitor_found;
 
-      // Save the mark, we might need it to extract the hash.
-      mv(tmp2_hash, tmp1_mark);
+    // Save the mark, we might need it to extract the hash.
+    mv(tmp2_hash, tmp1_mark);
 
-      // Look for the monitor in the current thread's object monitor cache (omc).
+    // Look for the monitor in the current thread's object monitor cache (omc).
 
-      ld(tmp1_monitor, Address(xthread, thr_omc_offset + omc_monitor_offset));
-      ld(tmp4, Address(xthread, thr_omc_offset + omc_obj_offset));
-      beq(obj, tmp4, monitor_found);
+    ld(tmp1_monitor, Address(xthread, thr_omc_offset + omc_monitor_offset));
+    ld(tmp4, Address(xthread, thr_omc_offset + omc_obj_offset));
+    beq(obj, tmp4, monitor_found);
 
-      // Look for the monitor in the table.
+    // Look for the monitor in the table.
 
-      // Get the hash code.
-      srli(tmp2_hash, tmp2_hash, markWord::hash_shift);
+    // Get the hash code.
+    srli(tmp2_hash, tmp2_hash, markWord::hash_shift);
 
-      // Get the table and calculate the bucket's address.
-      la(tmp3_t, ExternalAddress(ObjectMonitorTable::current_table_address()));
-      ld(tmp3_t, Address(tmp3_t));
-      ld(tmp1, Address(tmp3_t, ObjectMonitorTable::table_capacity_mask_offset()));
-      andr(tmp2_hash, tmp2_hash, tmp1);
-      ld(tmp3_t, Address(tmp3_t, ObjectMonitorTable::table_buckets_offset()));
+    // Get the table and calculate the bucket's address.
+    la(tmp3_t, ExternalAddress(ObjectMonitorTable::current_table_address()));
+    ld(tmp3_t, Address(tmp3_t));
+    ld(tmp1, Address(tmp3_t, ObjectMonitorTable::table_capacity_mask_offset()));
+    andr(tmp2_hash, tmp2_hash, tmp1);
+    ld(tmp3_t, Address(tmp3_t, ObjectMonitorTable::table_buckets_offset()));
 
-      // Read the monitor from the bucket.
-      shadd(tmp3_bucket, tmp2_hash, tmp3_t, tmp4, LogBytesPerWord);
-      ld(tmp1_monitor, Address(tmp3_bucket));
+    // Read the monitor from the bucket.
+    shadd(tmp3_bucket, tmp2_hash, tmp3_t, tmp4, LogBytesPerWord);
+    ld(tmp1_monitor, Address(tmp3_bucket));
 
-      // Check if the monitor in the bucket is special (empty, tombstone or removed).
-      mv(tmp2, ObjectMonitorTable::SpecialPointerValues::below_is_special);
-      bltu(tmp1_monitor, tmp2, slow_path);
+    // Check if the monitor in the bucket is special (empty, tombstone or removed).
+    mv(tmp2, ObjectMonitorTable::SpecialPointerValues::below_is_special);
+    bltu(tmp1_monitor, tmp2, slow_path);
 
-      // Check if object matches.
-      ld(tmp3, Address(tmp1_monitor, ObjectMonitor::object_offset()));
-      BarrierSetAssembler* bs_asm = BarrierSet::barrier_set()->barrier_set_assembler();
-      bs_asm->try_peek_weak_handle_in_nmethod(this, tmp3, tmp3, tmp2, slow_path);
-      bne(tmp3, obj, slow_path);
+    // Check if object matches.
+    ld(tmp3, Address(tmp1_monitor, ObjectMonitor::object_offset()));
+    BarrierSetAssembler* bs_asm = BarrierSet::barrier_set()->barrier_set_assembler();
+    bs_asm->try_peek_weak_handle_in_nmethod(this, tmp3, tmp3, tmp2, slow_path);
+    bne(tmp3, obj, slow_path);
 
-      // Store the monitor in the current thread's object monitor cache (omc).
-      sd(tmp1_monitor, Address(xthread, thr_omc_offset + omc_monitor_offset));
-      sd(obj, Address(xthread, thr_omc_offset + omc_obj_offset));
+    // Store the monitor in the current thread's object monitor cache (omc).
+    sd(tmp1_monitor, Address(xthread, thr_omc_offset + omc_monitor_offset));
+    sd(obj, Address(xthread, thr_omc_offset + omc_obj_offset));
 
-      bind(monitor_found);
-    }
+    bind(monitor_found);
 
     const Register tmp2_owner_addr = tmp2;
     const Register tmp3_owner = tmp3;
 
-    const ByteSize monitor_tag = in_ByteSize(UseObjectMonitorTable ? 0 : checked_cast<int>(markWord::monitor_value));
-    const Address owner_address(tmp1_monitor, ObjectMonitor::owner_offset() - monitor_tag);
-    const Address recursions_address(tmp1_monitor, ObjectMonitor::recursions_offset() - monitor_tag);
+    const Address owner_address(tmp1_monitor, ObjectMonitor::owner_offset());
+    const Address recursions_address(tmp1_monitor, ObjectMonitor::recursions_offset());
 
     Label monitor_locked;
 
@@ -224,10 +217,8 @@ void C2_MacroAssembler::fast_lock(Register obj, Register box,
     increment(recursions_address, 1, tmp2, tmp3);
 
     bind(monitor_locked);
-    if (UseObjectMonitorTable) {
-      // Cache the monitor for unlock.
-      sd(tmp1_monitor, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
-    }
+    // Cache the monitor for unlock.
+    sd(tmp1_monitor, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
   }
 
   bind(locked);
@@ -300,7 +291,7 @@ void C2_MacroAssembler::fast_unlock(Register obj, Register box,
     // there will be no monitor in the box. So we need to push back the obj
     // so that the runtime can fix any potential anonymous owner.
     test_bit(tmp3_t, tmp1_mark, exact_log2(markWord::monitor_value));
-    bnez(tmp3_t, UseObjectMonitorTable ? push_and_slow_path : inflated);
+    bnez(tmp3_t, push_and_slow_path);
 
     // Try to unlock. Transition lock bits 0b00 => 0b01
     assert(oopDesc::mark_offset_in_bytes() == 0, "required to avoid lea");
@@ -344,16 +335,10 @@ void C2_MacroAssembler::fast_unlock(Register obj, Register box,
 
     const Register tmp1_monitor = tmp1;
 
-    if (!UseObjectMonitorTable) {
-      assert(tmp1_monitor == tmp1_mark, "should be the same here");
-      // Untag the monitor.
-      subi(tmp1_monitor, tmp1_mark, (int)markWord::monitor_value);
-    } else {
-      ld(tmp1_monitor, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
-      // No valid pointer below alignof(ObjectMonitor*). Take the slow path.
-      mv(tmp3_t, alignof(ObjectMonitor*));
-      bltu(tmp1_monitor, tmp3_t, slow_path);
-    }
+    ld(tmp1_monitor, Address(box, BasicLock::object_monitor_cache_offset_in_bytes()));
+    // No valid pointer below alignof(ObjectMonitor*). Take the slow path.
+    mv(tmp3_t, alignof(ObjectMonitor*));
+    bltu(tmp1_monitor, tmp3_t, slow_path);
 
     const Register tmp2_recursions = tmp2;
     Label not_recursive;
@@ -423,6 +408,7 @@ void C2_MacroAssembler::fast_unlock(Register obj, Register box,
 // StringLatin1.indexOfChar
 void C2_MacroAssembler::string_indexof_char_short(Register str1, Register cnt1,
                                                   Register ch, Register result,
+                                                  Register start_index,
                                                   bool isL)
 {
   Register ch1 = t0;
@@ -515,7 +501,7 @@ void C2_MacroAssembler::string_indexof_char_short(Register str1, Register cnt1,
   addi(index, index, 7);
 
   bind(MATCH);
-  mv(result, index);
+  add(result, start_index, index);
   bind(NOMATCH);
   BLOCK_COMMENT("} string_indexof_char_short");
 }
@@ -528,39 +514,39 @@ void C2_MacroAssembler::string_indexof_char(Register str1, Register cnt1,
                                             Register tmp3, Register tmp4,
                                             bool isL)
 {
-  Label CH1_LOOP, HIT, NOMATCH, DONE, DO_LONG;
+  Label CH1_LOOP, HIT, NOMATCH, DONE, SHORT;
   Register ch1 = t0;
   Register orig_cnt = t1;
   Register mask1 = tmp3;
   Register mask2 = tmp2;
   Register match_mask = tmp1;
-  Register trailing_char = tmp4;
-  Register unaligned_elems = tmp4;
+  Register loop_step = tmp4;
+  Register trailing_chars = tmp4;
+  Register unaligned_chars = tmp4;
+  Register start_index = tmp4;
 
   BLOCK_COMMENT("string_indexof_char {");
   beqz(cnt1, NOMATCH);
 
   subi(t0, cnt1, isL ? 32 : 16);
-  bgtz(t0, DO_LONG);
-  string_indexof_char_short(str1, cnt1, ch, result, isL);
-  j(DONE);
+  mv(start_index, zr);
+  blez(t0, SHORT);
 
-  bind(DO_LONG);
   mv(orig_cnt, cnt1);
   if (AvoidUnalignedAccesses) {
     Label ALIGNED;
-    andi(unaligned_elems, str1, 0x7);
-    beqz(unaligned_elems, ALIGNED);
-    sub(unaligned_elems, unaligned_elems, 8);
-    neg(unaligned_elems, unaligned_elems);
+    andi(unaligned_chars, str1, 0x7);
+    beqz(unaligned_chars, ALIGNED);
+    sub(unaligned_chars, unaligned_chars, 8);
+    neg(unaligned_chars, unaligned_chars);
     if (!isL) {
-      srli(unaligned_elems, unaligned_elems, 1);
+      srli(unaligned_chars, unaligned_chars, 1);
     }
     // do unaligned part per element
-    string_indexof_char_short(str1, unaligned_elems, ch, result, isL);
+    string_indexof_char_short(str1, unaligned_chars, ch, result, zr, isL);
     bgez(result, DONE);
     mv(orig_cnt, cnt1);
-    sub(cnt1, cnt1, unaligned_elems);
+    sub(cnt1, cnt1, unaligned_chars);
     bind(ALIGNED);
   }
 
@@ -585,29 +571,48 @@ void C2_MacroAssembler::string_indexof_char(Register str1, Register cnt1,
   uint64_t mask7fff = UCONST64(0x7fff7fff7fff7fff);
   mv(mask2, isL ? mask7f7f : mask7fff);
 
+  mv(loop_step, 8);
+
   bind(CH1_LOOP);
   ld(ch1, Address(str1));
   addi(str1, str1, 8);
   subi(cnt1, cnt1, 8);
   compute_match_mask(ch1, ch, match_mask, mask1, mask2);
   bnez(match_mask, HIT);
-  bgtz(cnt1, CH1_LOOP);
-  j(NOMATCH);
+  bge(cnt1, loop_step, CH1_LOOP);
+
+  beqz(cnt1, NOMATCH);
+  if (!isL) {
+    srli(cnt1, cnt1, 1);
+  }
+  // Tail (1..7 chars) after the SWAR loop has advanced str1. cnt1 holds the
+  // remaining char count; the number of chars already scanned by the loop is
+  // (orig_cnt - cnt1). string_indexof_char_short returns an index relative to
+  // the current str1, so we pass that prefix as start_index to recover the
+  // real index.
+  // Note: ch was broadcast across all 8 bytes for the SWAR loop above, but the
+  // short helper compares a single element, so restore ch to a single char.
+  isL ? zext(ch, ch, 8) : zext(ch, ch, 16);
+  sub(start_index, orig_cnt, cnt1);
+
+  bind(SHORT);
+  string_indexof_char_short(str1, cnt1, ch, result, start_index, isL);
+  j(DONE);
 
   bind(HIT);
   // count bits of trailing zero chars
-  ctzc_bits(trailing_char, match_mask, isL, ch1, result);
-  srli(trailing_char, trailing_char, 3);
+  ctzc_bits(trailing_chars, match_mask, isL, ch1, result);
+  srli(trailing_chars, trailing_chars, 3);
   addi(cnt1, cnt1, 8);
-  ble(cnt1, trailing_char, NOMATCH);
+
   // match case
   if (!isL) {
     srli(cnt1, cnt1, 1);
-    srli(trailing_char, trailing_char, 1);
+    srli(trailing_chars, trailing_chars, 1);
   }
 
   sub(result, orig_cnt, cnt1);
-  add(result, result, trailing_char);
+  add(result, result, trailing_chars);
   j(DONE);
 
   bind(NOMATCH);
@@ -1852,7 +1857,7 @@ void C2_MacroAssembler::arrays_hashcode_v(Register ary, Register cnt, Register r
 
   vsetvli(consumed, cnt, Assembler::e32, Assembler::m2);
   vle32_v(v_coeffs, t1); // 31^^(stride - 1) ... 31^^0
-  vmv_v_x(v_sum, x0);
+  vmv_v_i(v_sum, 0);
 
   bind(VEC_LOOP);
   arrays_hashcode_elload_v(v_src, v_tmp, ary, eltype);
@@ -2036,47 +2041,49 @@ void C2_MacroAssembler::enc_cmpEqNe_imm0_branch(int cmpFlag, Register op1, Label
 }
 
 void C2_MacroAssembler::enc_cmove(int cmpFlag, Register op1, Register op2, Register dst, Register src) {
-  bool is_unsigned = (cmpFlag & unsigned_branch_mask) == unsigned_branch_mask;
-  int op_select = cmpFlag & (~unsigned_branch_mask);
+  if (dst != src) {
+    bool is_unsigned = (cmpFlag & unsigned_branch_mask) == unsigned_branch_mask;
+    int op_select = cmpFlag & (~unsigned_branch_mask);
 
-  switch (op_select) {
-    case BoolTest::eq:
-      cmov_eq(op1, op2, dst, src);
-      break;
-    case BoolTest::ne:
-      cmov_ne(op1, op2, dst, src);
-      break;
-    case BoolTest::le:
-      if (is_unsigned) {
-        cmov_leu(op1, op2, dst, src);
-      } else {
-        cmov_le(op1, op2, dst, src);
-      }
-      break;
-    case BoolTest::ge:
-      if (is_unsigned) {
-        cmov_geu(op1, op2, dst, src);
-      } else {
-        cmov_ge(op1, op2, dst, src);
-      }
-      break;
-    case BoolTest::lt:
-      if (is_unsigned) {
-        cmov_ltu(op1, op2, dst, src);
-      } else {
-        cmov_lt(op1, op2, dst, src);
-      }
-      break;
-    case BoolTest::gt:
-      if (is_unsigned) {
-        cmov_gtu(op1, op2, dst, src);
-      } else {
-        cmov_gt(op1, op2, dst, src);
-      }
-      break;
-    default:
-      assert(false, "unsupported compare condition");
-      ShouldNotReachHere();
+    switch (op_select) {
+      case BoolTest::eq:
+        cmov_eq(op1, op2, dst, src);
+        break;
+      case BoolTest::ne:
+        cmov_ne(op1, op2, dst, src);
+        break;
+      case BoolTest::le:
+        if (is_unsigned) {
+          cmov_leu(op1, op2, dst, src);
+        } else {
+          cmov_le(op1, op2, dst, src);
+        }
+        break;
+      case BoolTest::ge:
+        if (is_unsigned) {
+          cmov_geu(op1, op2, dst, src);
+        } else {
+          cmov_ge(op1, op2, dst, src);
+        }
+        break;
+      case BoolTest::lt:
+        if (is_unsigned) {
+          cmov_ltu(op1, op2, dst, src);
+        } else {
+          cmov_lt(op1, op2, dst, src);
+        }
+        break;
+      case BoolTest::gt:
+        if (is_unsigned) {
+          cmov_gtu(op1, op2, dst, src);
+        } else {
+          cmov_gt(op1, op2, dst, src);
+        }
+        break;
+      default:
+        assert(false, "unsupported compare condition");
+        ShouldNotReachHere();
+    }
   }
 }
 
@@ -2617,7 +2624,7 @@ void C2_MacroAssembler::java_round_float_v(VectorRegister dst, VectorRegister sr
   // replacing vfclass with feq as performance optimization
   vmfeq_vv(v0, src, src);
   // set dst = 0 in cases of NaN
-  vmv_v_x(dst, zr);
+  vmv_v_i(dst, 0);
 
   // dst = (src + 0.5) rounded down towards negative infinity
   vfadd_vf(dst, src, ftmp, Assembler::v0_t);
@@ -2641,7 +2648,7 @@ void C2_MacroAssembler::java_round_double_v(VectorRegister dst, VectorRegister s
   // replacing vfclass with feq as performance optimization
   vmfeq_vv(v0, src, src);
   // set dst = 0 in cases of NaN
-  vmv_v_x(dst, zr);
+  vmv_v_i(dst, 0);
 
   // dst = (src + 0.5) rounded down towards negative infinity
   vfadd_vf(dst, src, ftmp, Assembler::v0_t);
@@ -2699,7 +2706,7 @@ void C2_MacroAssembler::clear_array_v(Register base, Register cnt) {
 
   // making zero words
   vsetvli(t0, cnt, Assembler::e64, Assembler::m4);
-  vxor_vv(v4, v4, v4);
+  vmv_v_i(v4, 0);
 
   bind(loop);
   vsetvli(t0, cnt, Assembler::e64, Assembler::m4);
@@ -3296,7 +3303,7 @@ void C2_MacroAssembler::integer_narrow_v(VectorRegister dst, BasicType dst_bt, u
 #define VFCVT_SAFE(VFLOATCVT)                                                      \
 void C2_MacroAssembler::VFLOATCVT##_safe(VectorRegister dst, VectorRegister src) { \
   assert_different_registers(dst, src);                                            \
-  vxor_vv(dst, dst, dst);                                                          \
+  vmv_v_i(dst, 0);                                                                 \
   vmfeq_vv(v0, src, src);                                                          \
   VFLOATCVT(dst, src, Assembler::v0_t);                                            \
 }

src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.hpp

@@ -64,6 +64,7 @@
 
   void string_indexof_char_short(Register str1, Register cnt1,
                                  Register ch, Register result,
+                                 Register start_index,
                                  bool isL);
 
   void string_indexof_char(Register str1, Register cnt1,

src/hotspot/cpu/riscv/downcallLinker_riscv.cpp

@@ -308,7 +308,7 @@ void DowncallLinker::StubGenerator::generate() {
     __ restore_cpu_control_state_after_jni(t0);
 
     __ block_comment("{ thread native2java");
-    __ mv(t0, _thread_in_native_trans);
+    __ mv(t0, _thread_in_vm);
     __ sw(t0, Address(xthread, JavaThread::thread_state_offset()));
 
     // Force this write out before the read below
@@ -383,5 +383,5 @@ void DowncallLinker::StubGenerator::generate() {
 
   //////////////////////////////////////////////////////////////////////////////
 
-  __ flush();
+  // Code will be copied. No ICache sync required.
 }

src/hotspot/cpu/riscv/frame_riscv.cpp

@@ -629,8 +629,8 @@ intptr_t* frame::repair_sender_sp(nmethod* nm, intptr_t* sp, intptr_t** saved_fp
 }
 
 bool frame::was_augmented_on_entry(int& real_size) const {
-  assert(is_compiled_frame(), "");
-  assert(!_cb->as_nmethod_or_null()->needs_stack_repair(), "unimplemented");
+  assert(_cb != nullptr && _cb->is_nmethod(), "");
+  assert(!_cb->as_nmethod()->needs_stack_repair(), "unimplemented");
   real_size = _cb->frame_size();
   return false;
 }

src/hotspot/cpu/riscv/gc/g1/g1BarrierSetAssembler_riscv.cpp

@@ -24,6 +24,7 @@
  */
 
 #include "asm/macroAssembler.inline.hpp"
+#include "code/aotCodeCache.hpp"
 #include "gc/g1/g1BarrierSet.hpp"
 #include "gc/g1/g1BarrierSetAssembler.hpp"
 #include "gc/g1/g1BarrierSetRuntime.hpp"
@@ -118,13 +119,20 @@ void G1BarrierSetAssembler::gen_write_ref_array_post_barrier(MacroAssembler* mas
   // Iterate from start card to end card (inclusive).
   __ bind(loop);
   if (UseCondCardMark) {
+    // All non-clean cards (dirty, to-cset, from-remset) have bit0 == 0.
+    static_assert((G1CardTable::g1_dirty_card & 1U) == 0
+               && (G1CardTable::g1_to_cset_card & 1U) == 0
+               && (G1CardTable::g1_from_remset_card & 1U) == 0,
+               "cards needing scan must have bit0 == 0");
+    // Clean card has bit0 == 1.
+    static_assert(((uint)G1CardTable::clean_card_val() & 1U) == 1,
+                  "clean card must have bit0 == 1");
     __ lbu(tmp, Address(start, 0));
-    static_assert((uint)G1CardTable::clean_card_val() == 0xff, "must be");
-    __ subi(tmp, tmp, G1CardTable::clean_card_val()); // Convert to clean_card_value() to a comparison
-                                                      // against zero to avoid use of an extra temp.
-    __ bnez(tmp, next);
+    __ test_bit(tmp, tmp, 0);                         // test bit0: clean has bit0 == 1, non-clean has bit0 == 0
+    __ beqz(tmp, next);                               // skip store if already non-clean
   }
 
+  // `sb zr` writes 0, which must be the dirty value.
   static_assert(G1CardTable::dirty_card_val() == 0, "must be to use zr");
   __ sb(zr, Address(start, 0));
 
@@ -250,9 +258,22 @@ static void generate_post_barrier(MacroAssembler* masm,
   assert(thread == xthread, "must be");
   assert_different_registers(store_addr, new_val, thread, tmp1, tmp2, noreg);
   // Does store cross heap regions?
-  __ xorr(tmp1, store_addr, new_val);                    // tmp1 := store address ^ new value
-  __ srli(tmp1, tmp1, G1HeapRegion::LogOfHRGrainBytes);  // tmp1 := ((store address ^ new value) >> LogOfHRGrainBytes)
-  __ beqz(tmp1, done);
+#if INCLUDE_CDS
+  // AOT code needs to load the barrier grain shift from the aot
+  // runtime constants area in the code cache otherwise we can compile
+  // it as an immediate operand
+  if (AOTCodeCache::is_on_for_dump()) {
+    __ xorr(tmp1, store_addr, new_val);
+    __ lwu(tmp2, ExternalAddress(AOTRuntimeConstants::grain_shift_address()));
+    __ srl(tmp1, tmp1, tmp2);
+    __ beqz(tmp1, done);
+  } else
+#endif
+  {
+    __ xorr(tmp1, store_addr, new_val);                    // tmp1 := store address ^ new value
+    __ srli(tmp1, tmp1, G1HeapRegion::LogOfHRGrainBytes);  // tmp1 := ((store address ^ new value) >> LogOfHRGrainBytes)
+    __ beqz(tmp1, done);
+  }
 
   // Crosses regions, storing null?
   if (new_val_may_be_null) {
@@ -264,14 +285,23 @@ static void generate_post_barrier(MacroAssembler* masm,
   Address card_table_address(xthread, G1ThreadLocalData::card_table_base_offset());
   __ ld(tmp2, card_table_address);                       // tmp2 := card table base address
   __ add(tmp1, tmp1, tmp2);                              // tmp1 := card address
+
   if (UseCondCardMark) {
-    static_assert((uint)G1CardTable::clean_card_val() == 0xff, "must be");
+    // All non-clean cards (dirty, to-cset, from-remset) have bit0 == 0.
+    static_assert((G1CardTable::g1_dirty_card & 1U) == 0
+               && (G1CardTable::g1_to_cset_card & 1U) == 0
+               && (G1CardTable::g1_from_remset_card & 1U) == 0,
+               "cards needing scan must have bit0 == 0");
+    // Clean card has bit0 == 1.
+    static_assert(((uint)G1CardTable::clean_card_val() & 1U) == 1,
+                  "clean card must have bit0 == 1");
     __ lbu(tmp2, Address(tmp1, 0));                      // tmp2 := card
-    __ subi(tmp2, tmp2, G1CardTable::clean_card_val());  // Convert to clean_card_value() to a comparison
-                                                         // against zero to avoid use of an extra temp.
-    __ bnez(tmp2, done);
+    __ test_bit(tmp2, tmp2, 0);                          // test bit0: clean has bit0 == 1, non-clean has bit0 == 0
+    __ beqz(tmp2, done);                                 // skip store if already non-clean
   }
-  static_assert((uint)G1CardTable::dirty_card_val() == 0, "must be to use zr");
+
+  // `sb zr` writes 0, which must be the dirty value.
+  static_assert(G1CardTable::dirty_card_val() == 0, "must be to use zr");
   __ sb(zr, Address(tmp1, 0));
 }
 

src/hotspot/cpu/riscv/gc/shared/barrierSetAssembler_riscv.cpp

@@ -24,6 +24,7 @@
  */
 
 #include "classfile/classLoaderData.hpp"
+#include "code/aotCodeCache.hpp"
 #include "gc/shared/barrierSet.hpp"
 #include "gc/shared/barrierSetAssembler.hpp"
 #include "gc/shared/barrierSetNMethod.hpp"
@@ -300,8 +301,10 @@ void BarrierSetAssembler::nmethod_entry_barrier(MacroAssembler* masm, Label* slo
           // Because processors will not start the second load until the first comes back.
           // This means you can't overlap the two loads,
           // which is stronger than needed for ordering (stronger than TSO).
-          __ srli(ra, t0, 32);
-          __ orr(t1, t1, ra);
+          // XOR the guard into the epoch address twice. This preserves the
+          // address while making it dependent on the guard load.
+          __ xorr(t1, t1, t0);
+          __ xorr(t1, t1, t0);
         }
         // Read the global epoch value.
         __ lwu(t1, t1);
@@ -370,10 +373,20 @@ void BarrierSetAssembler::c2i_entry_barrier(MacroAssembler* masm) {
 }
 
 void BarrierSetAssembler::check_oop(MacroAssembler* masm, Register obj, Register tmp1, Register tmp2, Label& error) {
+  assert_different_registers(obj, tmp1, tmp2);
   // Check if the oop is in the right area of memory
-  __ mv(tmp2, (intptr_t) Universe::verify_oop_mask());
-  __ andr(tmp1, obj, tmp2);
-  __ mv(tmp2, (intptr_t) Universe::verify_oop_bits());
+#if INCLUDE_CDS
+  if (AOTCodeCache::is_on_for_dump()) {
+    __ ld(tmp2, ExternalAddress(AOTRuntimeConstants::verify_oop_mask_address()));
+    __ andr(tmp1, obj, tmp2);
+    __ ld(tmp2, ExternalAddress(AOTRuntimeConstants::verify_oop_bits_address()));
+  } else
+#endif
+  {
+    __ mv(tmp2, (intptr_t) Universe::verify_oop_mask());
+    __ andr(tmp1, obj, tmp2);
+    __ mv(tmp2, (intptr_t) Universe::verify_oop_bits());
+  }
 
   // Compare tmp1 and tmp2.
   __ bne(tmp1, tmp2, error);

src/hotspot/cpu/riscv/gc/shenandoah/shenandoahBarrierSetAssembler_riscv.cpp

@@ -24,6 +24,7 @@
  *
  */
 
+#include "code/aotCodeCache.hpp"
 #include "gc/shenandoah/heuristics/shenandoahHeuristics.hpp"
 #include "gc/shenandoah/mode/shenandoahMode.hpp"
 #include "gc/shenandoah/shenandoahBarrierSet.hpp"
@@ -219,8 +220,17 @@ void ShenandoahBarrierSetAssembler::load_reference_barrier(MacroAssembler* masm,
 
   // Test for in-cset
   if (is_strong) {
-    __ mv(t1, ShenandoahHeap::in_cset_fast_test_addr());
-    __ srli(t0, x10, ShenandoahHeapRegion::region_size_bytes_shift_jint());
+#if INCLUDE_CDS
+    if (AOTCodeCache::is_on_for_dump()) {
+      __ ld(t1, ExternalAddress(AOTRuntimeConstants::cset_base_address()));
+      __ lwu(t0, ExternalAddress(AOTRuntimeConstants::grain_shift_address()));
+      __ srl(t0, x10, t0);
+    } else
+#endif
+    {
+      __ mv(t1, ShenandoahHeap::in_cset_fast_test_addr());
+      __ srli(t0, x10, ShenandoahHeapRegion::region_size_bytes_shift_jint());
+    }
     __ add(t1, t1, t0);
     __ lbu(t1, Address(t1));
     __ test_bit(t0, t1, 0);
@@ -434,10 +444,20 @@ void ShenandoahBarrierSetAssembler::try_peek_weak_handle_in_nmethod(MacroAssembl
 }
 
 void ShenandoahBarrierSetAssembler::check_oop(MacroAssembler* masm, Register obj, Register tmp1, Register tmp2, Label& L_error) {
+  assert_different_registers(obj, tmp1, tmp2);
   // Check if the oop is in the right area of memory
-  __ mv(tmp2, (intptr_t) Universe::verify_oop_mask());
-  __ andr(tmp1, obj, tmp2);
-  __ mv(tmp2, (intptr_t) Universe::verify_oop_bits());
+#if INCLUDE_CDS
+  if (AOTCodeCache::is_on_for_dump()) {
+    __ ld(tmp2, ExternalAddress(AOTRuntimeConstants::verify_oop_mask_address()));
+    __ andr(tmp1, obj, tmp2);
+    __ ld(tmp2, ExternalAddress(AOTRuntimeConstants::verify_oop_bits_address()));
+  } else
+#endif
+  {
+    __ mv(tmp2, (intptr_t) Universe::verify_oop_mask());
+    __ andr(tmp1, obj, tmp2);
+    __ mv(tmp2, (intptr_t) Universe::verify_oop_bits());
+  }
 
   // Compare tmp1 and tmp2.
   __ bne(tmp1, tmp2, L_error);
@@ -815,8 +835,14 @@ void ShenandoahBarrierStubC2::lrb(MacroAssembler& masm) {
     __ mv(_tmp2, _obj);
   }
 
-  __ mv(_tmp1, ShenandoahHeap::in_cset_fast_test_addr());
-  __ srli(_tmp2, _tmp2, ShenandoahHeapRegion::region_size_bytes_shift_jint());
+  if (AOTCodeCache::is_on_for_dump()) {
+    __ lwu(_tmp1, ExternalAddress(AOTRuntimeConstants::grain_shift_address()));
+    __ srl(_tmp2, _tmp2, _tmp1);
+    __ ld(_tmp1, ExternalAddress(AOTRuntimeConstants::cset_base_address()));
+  } else {
+    __ mv(_tmp1, ShenandoahHeap::in_cset_fast_test_addr());
+    __ srli(_tmp2, _tmp2, ShenandoahHeapRegion::region_size_bytes_shift_jint());
+  }
   __ add(_tmp1, _tmp1, _tmp2);
   __ lbu(_tmp1, Address(_tmp1, 0));
   maybe_far_jump_if_zero(masm, _tmp1);
@@ -854,7 +880,7 @@ void ShenandoahBarrierStubC2::lrb(MacroAssembler& masm) {
     // Save the result where needed. Narrow entries return narrowOop (32 bits)
     // we need to zero the upper 32 bits of x10.
     if (_narrow) {
-      __ zext_w(_obj, x10);
+      __ zext(_obj, x10, 32);
     } else {
       __ mv(_obj, x10);
     }

src/hotspot/cpu/riscv/gc/z/zBarrierSetAssembler_riscv.cpp

@@ -24,6 +24,7 @@
  */
 
 #include "asm/macroAssembler.inline.hpp"
+#include "code/aotCodeCache.hpp"
 #include "code/codeBlob.hpp"
 #include "code/vmreg.inline.hpp"
 #include "gc/z/zAddress.hpp"
@@ -1007,6 +1008,7 @@ void ZBarrierSetAssembler::generate_c1_store_barrier_stub(LIR_Assembler* ce,
 #define __ masm->
 
 void ZBarrierSetAssembler::check_oop(MacroAssembler* masm, Register obj, Register tmp1, Register tmp2, Label& error) {
+  assert_different_registers(obj, tmp1, tmp2);
   // C1 calls verify_oop in the middle of barriers, before they have been uncolored
   // and after being colored. Therefore, we must deal with colored oops as well.
   Label done;
@@ -1044,9 +1046,18 @@ void ZBarrierSetAssembler::check_oop(MacroAssembler* masm, Register obj, Registe
 
   __ bind(check_zaddress);
   // Check if the oop is the right area of memory
-  __ mv(tmp1, (intptr_t) Universe::verify_oop_mask());
-  __ andr(tmp1, tmp1, obj);
-  __ mv(obj, (intptr_t) Universe::verify_oop_bits());
+#if INCLUDE_CDS
+  if (AOTCodeCache::is_on_for_dump()) {
+    __ ld(tmp1, ExternalAddress(AOTRuntimeConstants::verify_oop_mask_address()));
+    __ andr(tmp1, tmp1, obj);
+    __ ld(obj, ExternalAddress(AOTRuntimeConstants::verify_oop_bits_address()));
+  } else
+#endif
+  {
+    __ mv(tmp1, (intptr_t) Universe::verify_oop_mask());
+    __ andr(tmp1, tmp1, obj);
+    __ mv(obj, (intptr_t) Universe::verify_oop_bits());
+  }
   __ bne(tmp1, obj, error);
 
   __ bind(done);

src/hotspot/cpu/riscv/gc/z/z_riscv.ad

@@ -33,7 +33,7 @@ source_hpp %{
 source %{
 #include "gc/z/zBarrierSetAssembler.hpp"
 
-static void z_color(MacroAssembler* masm, const MachNode* node, Register dst, Register src, Register tmp) {
+static void z_color(MacroAssembler* masm, Register dst, Register src, Register tmp) {
   assert_different_registers(dst, tmp);
 
   __ relocate(barrier_Relocation::spec(), [&] {
@@ -43,7 +43,7 @@ static void z_color(MacroAssembler* masm, const MachNode* node, Register dst, Re
   __ orr(dst, dst, tmp);
 }
 
-static void z_uncolor(MacroAssembler* masm, const MachNode* node, Register ref) {
+static void z_uncolor(MacroAssembler* masm, Register ref) {
   __ srli(ref, ref, ZPointerLoadShift);
 }
 
@@ -63,7 +63,7 @@ static void z_load_barrier(MacroAssembler* masm, const MachNode* node, Address r
       ((node->barrier_data() & ZBarrierPhantom) != 0);
 
   if (node->barrier_data() == ZBarrierElided) {
-    z_uncolor(masm, node, ref);
+    z_uncolor(masm, ref);
     return;
   }
 
@@ -74,14 +74,14 @@ static void z_load_barrier(MacroAssembler* masm, const MachNode* node, Address r
   __ j(*stub->entry());
 
   __ bind(good);
-  z_uncolor(masm, node, ref);
+  z_uncolor(masm, ref);
   __ bind(*stub->continuation());
 }
 
 static void z_store_barrier(MacroAssembler* masm, const MachNode* node, Address ref_addr, Register rnew_zaddress, Register rnew_zpointer, Register tmp, bool is_atomic) {
   Assembler::InlineSkippedInstructionsCounter skipped_counter(masm);
   if (node->barrier_data() == ZBarrierElided) {
-    z_color(masm, node, rnew_zpointer, rnew_zaddress, tmp);
+    z_color(masm, rnew_zpointer, rnew_zaddress, tmp);
   } else {
     bool is_native = (node->barrier_data() & ZBarrierNative) != 0;
     bool is_nokeepalive = (node->barrier_data() & ZBarrierNoKeepalive) != 0;
@@ -145,7 +145,7 @@ instruct zCompareAndSwapP(iRegINoSp res, indirect mem, iRegP oldval, iRegP newva
   ins_encode %{
     guarantee($mem$$disp == 0, "impossible encoding");
     Address ref_addr($mem$$Register);
-    z_color(masm, this, $oldval_tmp$$Register, $oldval$$Register, $tmp1$$Register);
+    z_color(masm, $oldval_tmp$$Register, $oldval$$Register, $tmp1$$Register);
     z_store_barrier(masm, this, ref_addr, $newval$$Register, $newval_tmp$$Register, $tmp1$$Register, true /* is_atomic */);
     __ cmpxchg($mem$$Register, $oldval_tmp$$Register, $newval_tmp$$Register, Assembler::int64, Assembler::relaxed /* acquire */, Assembler::rl /* release */, $res$$Register, true /* result_as_bool */);
   %}
@@ -168,7 +168,7 @@ instruct zCompareAndSwapPAcq(iRegINoSp res, indirect mem, iRegP oldval, iRegP ne
   ins_encode %{
     guarantee($mem$$disp == 0, "impossible encoding");
     Address ref_addr($mem$$Register);
-    z_color(masm, this, $oldval_tmp$$Register, $oldval$$Register, $tmp1$$Register);
+    z_color(masm, $oldval_tmp$$Register, $oldval$$Register, $tmp1$$Register);
     z_store_barrier(masm, this, ref_addr, $newval$$Register, $newval_tmp$$Register, $tmp1$$Register, true /* is_atomic */);
     __ cmpxchg($mem$$Register, $oldval_tmp$$Register, $newval_tmp$$Register, Assembler::int64, Assembler::aq /* acquire */, Assembler::rl /* release */, $res$$Register, true /* result_as_bool */);
   %}
@@ -189,10 +189,10 @@ instruct zCompareAndExchangeP(iRegPNoSp res, indirect mem, iRegP oldval, iRegP n
   ins_encode %{
     guarantee($mem$$disp == 0, "impossible encoding");
     Address ref_addr($mem$$Register);
-    z_color(masm, this, $oldval_tmp$$Register, $oldval$$Register, $tmp1$$Register);
+    z_color(masm, $oldval_tmp$$Register, $oldval$$Register, $tmp1$$Register);
     z_store_barrier(masm, this, ref_addr, $newval$$Register, $newval_tmp$$Register, $tmp1$$Register, true /* is_atomic */);
     __ cmpxchg($mem$$Register, $oldval_tmp$$Register, $newval_tmp$$Register, Assembler::int64, Assembler::relaxed /* acquire */, Assembler::rl /* release */, $res$$Register);
-    z_uncolor(masm, this, $res$$Register);
+    z_uncolor(masm, $res$$Register);
   %}
 
   ins_pipe(pipe_slow);
@@ -211,10 +211,10 @@ instruct zCompareAndExchangePAcq(iRegPNoSp res, indirect mem, iRegP oldval, iReg
   ins_encode %{
     guarantee($mem$$disp == 0, "impossible encoding");
     Address ref_addr($mem$$Register);
-    z_color(masm, this, $oldval_tmp$$Register, $oldval$$Register, $tmp1$$Register);
+    z_color(masm, $oldval_tmp$$Register, $oldval$$Register, $tmp1$$Register);
     z_store_barrier(masm, this, ref_addr, $newval$$Register, $newval_tmp$$Register, $tmp1$$Register, true /* is_atomic */);
     __ cmpxchg($mem$$Register, $oldval_tmp$$Register, $newval_tmp$$Register, Assembler::int64, Assembler::aq /* acquire */, Assembler::rl /* release */, $res$$Register);
-    z_uncolor(masm, this, $res$$Register);
+    z_uncolor(masm, $res$$Register);
   %}
 
   ins_pipe(pipe_slow);
@@ -232,7 +232,7 @@ instruct zGetAndSetP(indirect mem, iRegP newv, iRegPNoSp prev, iRegPNoSp tmp, rF
   ins_encode %{
     z_store_barrier(masm, this, Address($mem$$Register), $newv$$Register, $prev$$Register, $tmp$$Register, true /* is_atomic */);
     __ atomic_xchg($prev$$Register, $prev$$Register, $mem$$Register);
-    z_uncolor(masm, this, $prev$$Register);
+    z_uncolor(masm, $prev$$Register);
   %}
 
   ins_pipe(pipe_serial);
@@ -250,7 +250,7 @@ instruct zGetAndSetPAcq(indirect mem, iRegP newv, iRegPNoSp prev, iRegPNoSp tmp,
   ins_encode %{
     z_store_barrier(masm, this, Address($mem$$Register), $newv$$Register, $prev$$Register, $tmp$$Register, true /* is_atomic */);
     __ atomic_xchgal($prev$$Register, $prev$$Register, $mem$$Register);
-    z_uncolor(masm, this, $prev$$Register);
+    z_uncolor(masm, $prev$$Register);
   %}
   ins_pipe(pipe_serial);
 %}

src/hotspot/cpu/riscv/interpreterRT_riscv.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2003, 2025, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2026, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2014, 2020, Red Hat Inc. All rights reserved.
  * Copyright (c) 2020, 2022, Huawei Technologies Co., Ltd. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
@@ -168,7 +168,7 @@ void InterpreterRuntime::SignatureHandlerGenerator::generate(uint64_t fingerprin
   __ movptr(x10, ExternalAddress(Interpreter::result_handler(method()->result_type())));
   __ ret();
 
-  __ flush();
+  __ invalidate_icache();
 }
 
 

src/hotspot/cpu/riscv/jniFastGetField_riscv.cpp

@@ -166,7 +166,7 @@ address JNI_FastGetField::generate_fast_get_int_field0(BasicType type) {
     __ leave();
     __ ret();
   }
-  __ flush();
+  __ invalidate_icache();
 
   return fast_entry;
 }

src/hotspot/cpu/riscv/macroAssembler_riscv.cpp

@@ -26,7 +26,9 @@
 
 #include "asm/assembler.hpp"
 #include "asm/assembler.inline.hpp"
+#include "cds/archiveBuilder.hpp"
 #include "ci/ciInlineKlass.hpp"
+#include "code/aotCodeCache.hpp"
 #include "code/compiledIC.hpp"
 #include "compiler/disassembler.hpp"
 #include "gc/shared/barrierSet.hpp"
@@ -160,8 +162,7 @@ uint32_t MacroAssembler::get_membar_kind(address addr) {
   assert_cond(addr != nullptr);
   assert(is_membar(addr), "no membar found");
 
-  uint32_t insn = Bytes::get_native_u4(addr);
-
+  uint32_t insn = Assembler::ld_instr(addr);
   uint32_t predecessor = Assembler::extract(insn, 27, 24);
   uint32_t successor = Assembler::extract(insn, 23, 20);
 
@@ -177,7 +178,7 @@ void MacroAssembler::set_membar_kind(address addr, uint32_t order_kind) {
 
   MacroAssembler::membar_mask_to_pred_succ(order_kind, predecessor, successor);
 
-  uint32_t insn = Bytes::get_native_u4(addr);
+  uint32_t insn = Assembler::ld_instr(addr);
   address pInsn = (address) &insn;
   Assembler::patch(pInsn, 27, 24, predecessor);
   Assembler::patch(pInsn, 23, 20, successor);
@@ -522,20 +523,25 @@ void MacroAssembler::_verify_oop(Register reg, const char* s, const char* file,
     ResourceMark rm;
     stringStream ss;
     ss.print("verify_oop: %s: %s (%s:%d)", reg->name(), s, file, line);
-    b = code_string(ss.as_string());
+#if INCLUDE_CDS
+    if (AOTCodeCache::is_on_for_dump() && !code_section()->scratch_emit()) {
+      // This will duplicate string to preserve it.
+      b = AOTCodeCache::add_C_string(ss.as_string());
+    } else
+#endif
+    {
+      b = code_string(ss.as_string());
+    }
   }
   BLOCK_COMMENT("verify_oop {");
 
   push_reg(RegSet::of(ra, t0, t1, c_rarg0), sp);
 
   mv(c_rarg0, reg); // c_rarg0 : x10
-  {
-    // The length of the instruction sequence emitted should not depend
-    // on the address of the char buffer so that the size of mach nodes for
-    // scratch emit and normal emit matches.
-    IncompressibleScope scope(this); // Fixed length
-    movptr(t0, (address) b);
-  }
+  // The length of the instruction sequence emitted should not depend
+  // on the address of the char buffer so that the size of mach nodes for
+  // scratch emit and normal emit matches.
+  la(t0, ExternalAddress((address)b));
 
   // Call indirectly to solve generation ordering problem
   ld(t1, RuntimeAddress(StubRoutines::verify_oop_subroutine_entry_address()));
@@ -708,7 +714,15 @@ void MacroAssembler::_verify_oop_addr(Address addr, const char* s, const char* f
     ResourceMark rm;
     stringStream ss;
     ss.print("verify_oop_addr: %s (%s:%d)", s, file, line);
-    b = code_string(ss.as_string());
+#if INCLUDE_CDS
+    if (AOTCodeCache::is_on_for_dump() && !code_section()->scratch_emit()) {
+      // This will duplicate string to preserve it.
+      b = AOTCodeCache::add_C_string(ss.as_string());
+    } else
+#endif
+    {
+      b = code_string(ss.as_string());
+    }
   }
   BLOCK_COMMENT("verify_oop_addr {");
 
@@ -721,13 +735,10 @@ void MacroAssembler::_verify_oop_addr(Address addr, const char* s, const char* f
     ld(x10, addr);
   }
 
-  {
-    // The length of the instruction sequence emitted should not depend
-    // on the address of the char buffer so that the size of mach nodes for
-    // scratch emit and normal emit matches.
-    IncompressibleScope scope(this); // Fixed length
-    movptr(t0, (address) b);
-  }
+  // The length of the instruction sequence emitted should not depend
+  // on the address of the char buffer so that the size of mach nodes for
+  // scratch emit and normal emit matches.
+  la(t0, ExternalAddress((address)b));
 
   // Call indirectly to solve generation ordering problem
   ld(t1, RuntimeAddress(StubRoutines::verify_oop_subroutine_entry_address()));
@@ -879,9 +890,13 @@ void MacroAssembler::resolve_global_jobject(Register value, Register tmp1, Regis
 }
 
 void MacroAssembler::stop(const char* msg) {
-  BLOCK_COMMENT(msg);
+  // Skip AOT caching C strings in scratch buffer.
+  const char* str = (code_section()->scratch_emit()) ? msg : AOTCodeCache::add_C_string(msg);
+  BLOCK_COMMENT(str);
+  // load msg into a0 so we can access it from the signal handler
+  // ExternalAddress enables saving and restoring via the code cache
+  la(c_rarg0, ExternalAddress((address) str));
   illegal_instruction(Assembler::csr::time);
-  emit_int64((uintptr_t)msg);
 }
 
 void MacroAssembler::unimplemented(const char* what) {
@@ -911,10 +926,8 @@ void MacroAssembler::emit_static_call_stub() {
 void MacroAssembler::call_VM_leaf_base(address entry_point,
                                        int number_of_arguments,
                                        Label *retaddr) {
-  int32_t offset = 0;
   push_reg(RegSet::of(t1, xmethod), sp);   // push << t1 & xmethod >> to sp
-  movptr(t1, entry_point, offset, t0);
-  jalr(t1, offset);
+  rt_call(entry_point, t1, t0);
   if (retaddr != nullptr) {
     bind(*retaddr);
   }
@@ -1164,16 +1177,17 @@ void MacroAssembler::jalr(Register Rs, int32_t offset) {
   Assembler::jalr(x1, Rs, offset);
 }
 
-void MacroAssembler::rt_call(address dest, Register tmp) {
-  assert(tmp != x5, "tmp register must not be x5.");
+void MacroAssembler::rt_call(address dest, Register tmp1, Register tmp2) {
+  assert_different_registers(tmp1, x5);
+  assert_different_registers(tmp1, tmp2);
   RuntimeAddress target(dest);
   if (CodeCache::contains(dest)) {
-    far_call(target, tmp);
+    far_call(target, tmp1);
   } else {
     relocate(target.rspec(), [&] {
       int32_t offset;
-      movptr(tmp, target.target(), offset);
-      jalr(tmp, offset);
+      movptr(tmp1, target.target(), offset, tmp2);
+      jalr(tmp1, offset);
     });
   }
 }
@@ -1251,13 +1265,33 @@ void MacroAssembler::wrap_label(Register r1, Register r2, Label &L,
 
 #undef INSN
 
-// cmov
+// cmov_zicond_eqz: dst = (cond == 0) ? src : dst
+void MacroAssembler::cmov_zicond_eqz(Register dst, Register src, Register cond, Register tmp) {
+  assert(UseZicond, "UseZicond must be enabled");
+  assert_different_registers(dst, src, cond);
+  czero_eqz(dst, dst, cond);
+  if (src != zr) {
+    czero_nez(tmp, src, cond);
+    add(dst, dst, tmp);
+  }
+}
+
+// cmov_zicond_nez: dst = (cond != 0) ? src : dst
+void MacroAssembler::cmov_zicond_nez(Register dst, Register src, Register cond, Register tmp) {
+  assert(UseZicond, "UseZicond must be enabled");
+  assert_different_registers(dst, src, cond);
+  czero_nez(dst, dst, cond);
+  if (src != zr) {
+    czero_eqz(tmp, src, cond);
+    add(dst, dst, tmp);
+  }
+}
+
 void MacroAssembler::cmov_eq(Register cmp1, Register cmp2, Register dst, Register src) {
+  assert_different_registers(dst, src);
   if (UseZicond) {
     xorr(t0, cmp1, cmp2);
-    czero_eqz(dst, dst, t0);
-    czero_nez(t0 , src, t0);
-    orr(dst, dst, t0);
+    cmov_zicond_eqz(dst, src, t0, t0);
     return;
   }
   Label no_set;
@@ -1267,11 +1301,10 @@ void MacroAssembler::cmov_eq(Register cmp1, Register cmp2, Register dst, Registe
 }
 
 void MacroAssembler::cmov_ne(Register cmp1, Register cmp2, Register dst, Register src) {
+  assert_different_registers(dst, src);
   if (UseZicond) {
     xorr(t0, cmp1, cmp2);
-    czero_nez(dst, dst, t0);
-    czero_eqz(t0 , src, t0);
-    orr(dst, dst, t0);
+    cmov_zicond_nez(dst, src, t0, t0);
     return;
   }
   Label no_set;
@@ -1281,11 +1314,10 @@ void MacroAssembler::cmov_ne(Register cmp1, Register cmp2, Register dst, Registe
 }
 
 void MacroAssembler::cmov_le(Register cmp1, Register cmp2, Register dst, Register src) {
+  assert_different_registers(dst, src);
   if (UseZicond) {
     slt(t0, cmp2, cmp1);
-    czero_eqz(dst, dst, t0);
-    czero_nez(t0,  src, t0);
-    orr(dst, dst, t0);
+    cmov_zicond_eqz(dst, src, t0, t0);
     return;
   }
   Label no_set;
@@ -1295,11 +1327,10 @@ void MacroAssembler::cmov_le(Register cmp1, Register cmp2, Register dst, Registe
 }
 
 void MacroAssembler::cmov_leu(Register cmp1, Register cmp2, Register dst, Register src) {
+  assert_different_registers(dst, src);
   if (UseZicond) {
     sltu(t0, cmp2, cmp1);
-    czero_eqz(dst, dst, t0);
-    czero_nez(t0,  src, t0);
-    orr(dst, dst, t0);
+    cmov_zicond_eqz(dst, src, t0, t0);
     return;
   }
   Label no_set;
@@ -1309,11 +1340,10 @@ void MacroAssembler::cmov_leu(Register cmp1, Register cmp2, Register dst, Regist
 }
 
 void MacroAssembler::cmov_ge(Register cmp1, Register cmp2, Register dst, Register src) {
+  assert_different_registers(dst, src);
   if (UseZicond) {
     slt(t0, cmp1, cmp2);
-    czero_eqz(dst, dst, t0);
-    czero_nez(t0,  src, t0);
-    orr(dst, dst, t0);
+    cmov_zicond_eqz(dst, src, t0, t0);
     return;
   }
   Label no_set;
@@ -1323,11 +1353,10 @@ void MacroAssembler::cmov_ge(Register cmp1, Register cmp2, Register dst, Registe
 }
 
 void MacroAssembler::cmov_geu(Register cmp1, Register cmp2, Register dst, Register src) {
+  assert_different_registers(dst, src);
   if (UseZicond) {
     sltu(t0, cmp1, cmp2);
-    czero_eqz(dst, dst, t0);
-    czero_nez(t0,  src, t0);
-    orr(dst, dst, t0);
+    cmov_zicond_eqz(dst, src, t0, t0);
     return;
   }
   Label no_set;
@@ -1337,11 +1366,10 @@ void MacroAssembler::cmov_geu(Register cmp1, Register cmp2, Register dst, Regist
 }
 
 void MacroAssembler::cmov_lt(Register cmp1, Register cmp2, Register dst, Register src) {
+  assert_different_registers(dst, src);
   if (UseZicond) {
     slt(t0, cmp1, cmp2);
-    czero_nez(dst, dst, t0);
-    czero_eqz(t0,  src, t0);
-    orr(dst, dst, t0);
+    cmov_zicond_nez(dst, src, t0, t0);
     return;
   }
   Label no_set;
@@ -1351,11 +1379,10 @@ void MacroAssembler::cmov_lt(Register cmp1, Register cmp2, Register dst, Registe
 }
 
 void MacroAssembler::cmov_ltu(Register cmp1, Register cmp2, Register dst, Register src) {
+  assert_different_registers(dst, src);
   if (UseZicond) {
     sltu(t0, cmp1, cmp2);
-    czero_nez(dst, dst, t0);
-    czero_eqz(t0,  src, t0);
-    orr(dst, dst, t0);
+    cmov_zicond_nez(dst, src, t0, t0);
     return;
   }
   Label no_set;
@@ -1365,11 +1392,10 @@ void MacroAssembler::cmov_ltu(Register cmp1, Register cmp2, Register dst, Regist
 }
 
 void MacroAssembler::cmov_gt(Register cmp1, Register cmp2, Register dst, Register src) {
+  assert_different_registers(dst, src);
   if (UseZicond) {
     slt(t0, cmp2, cmp1);
-    czero_nez(dst, dst, t0);
-    czero_eqz(t0,  src, t0);
-    orr(dst, dst, t0);
+    cmov_zicond_nez(dst, src, t0, t0);
     return;
   }
   Label no_set;
@@ -1379,11 +1405,10 @@ void MacroAssembler::cmov_gt(Register cmp1, Register cmp2, Register dst, Registe
 }
 
 void MacroAssembler::cmov_gtu(Register cmp1, Register cmp2, Register dst, Register src) {
+  assert_different_registers(dst, src);
   if (UseZicond) {
     sltu(t0, cmp2, cmp1);
-    czero_nez(dst, dst, t0);
-    czero_eqz(t0,  src, t0);
-    orr(dst, dst, t0);
+    cmov_zicond_nez(dst, src, t0, t0);
     return;
   }
   Label no_set;
@@ -2182,7 +2207,7 @@ void MacroAssembler::vector_update_crc32(Register crc, Register buf, Register le
       vsetivli(zr, N, Assembler::e32, Assembler::m1, Assembler::mu, Assembler::tu);
     }
 
-    vmv_v_x(vcrc, zr);
+    vmv_v_i(vcrc, 0);
     vmv_s_x(vcrc, crc);
 
     // multiple of 64
@@ -2327,7 +2352,7 @@ void MacroAssembler::kernel_crc32_vclmul_fold_vectorsize_16(Register crc, Regist
   vle64_v(v6, buf); addi(buf, buf, STEP);
   vle64_v(v7, buf); addi(buf, buf, STEP);
 
-  vmv_v_x(v31, zr);
+  vmv_v_i(v31, 0);
   vsetivli(zr, 1, Assembler::e32, Assembler::m1, Assembler::mu, Assembler::tu);
   vmv_s_x(v31, crc);
   vsetivli(zr, N, Assembler::e64, Assembler::m1, Assembler::mu, Assembler::tu);
@@ -2450,7 +2475,7 @@ void MacroAssembler::kernel_crc32_vclmul_fold_vectorsize_32(Register crc, Regist
   //    now, v1 should contains: 010101...
 
   // initial crc
-  vmv_v_x(v24, zr);
+  vmv_v_i(v24, 0);
   vsetivli(zr, 1, Assembler::e32, Assembler::m4, Assembler::mu, Assembler::tu);
   vmv_s_x(v24, crc);
   vsetivli(zr, N, Assembler::e64, Assembler::m4, Assembler::mu, Assembler::tu);
@@ -3054,7 +3079,7 @@ int MacroAssembler::patch_oop(address insn_addr, address o) {
 
 void MacroAssembler::reinit_heapbase() {
   if (UseCompressedOops) {
-    if (Universe::is_fully_initialized()) {
+    if (Universe::is_fully_initialized() && !AOTCodeCache::is_on_for_dump()) {
       mv(xheapbase, CompressedOops::base());
     } else {
       ld(xheapbase, ExternalAddress(CompressedOops::base_addr()));
@@ -3765,16 +3790,6 @@ void MacroAssembler::test_oop_prototype_bit(Register oop, Register temp_reg, int
   assert_different_registers(temp_reg, t0);
   // load mark word
   ld(temp_reg, Address(oop, oopDesc::mark_offset_in_bytes()));
-  if (!UseObjectMonitorTable) {
-    Label test_mark_word;
-    // check displaced
-    test_bit(t0, temp_reg, exact_log2(markWord::unlocked_value));
-    bnez(t0, test_mark_word);
-    // slow path use klass prototype
-    load_prototype_header(temp_reg, oop);
-
-    bind(test_mark_word);
-  }
   andi(temp_reg, temp_reg, tst_bit);
   if (jmp_set) {
     bnez(temp_reg, jmp_label, /* is_far */ true);
@@ -3828,11 +3843,17 @@ void MacroAssembler::encode_heap_oop(Register d, Register s) {
       mv(d, s);
     }
   } else {
-    Label notNull;
-    sub(d, s, xheapbase);
-    bgez(d, notNull);
-    mv(d, zr);
-    bind(notNull);
+    if (UseZicond) {
+      assert_different_registers(s, t0);
+      sub(t0, s, xheapbase);
+      czero_eqz(d, t0, s);  // d = s == 0 ? 0 : t0
+    } else {
+      Label notNull;
+      sub(d, s, xheapbase);
+      bgez(d, notNull);
+      mv(d, zr);
+      bind(notNull);
+    }
     if (CompressedOops::shift() != 0) {
       assert (LogMinObjAlignmentInBytes == CompressedOops::shift(), "decode alg wrong");
       srli(d, d, CompressedOops::shift());
@@ -3935,19 +3956,28 @@ void MacroAssembler::decode_klass_not_null(Register dst, Register src, Register
   assert_different_registers(dst, tmp);
   assert_different_registers(src, tmp);
 
-  if (CompressedKlassPointers::base() == nullptr) {
+  Register xbase = tmp;
+
+  if (AOTCodeCache::is_on_for_dump()) {
+    // We are generating code during AOT buildup that will run in *future* processes
+    // with likely different encoding settings. Therefore, we have to load the
+    // encoding base dynamically, we cannot just bake it in as immediate.
+    // Note that we only need to do this for base. The encoding shift would be the
+    // same between build time and runtime: the standard precomputed shift.
+    assert(CompressedKlassPointers::shift() == ArchiveBuilder::precomputed_narrow_klass_shift(),
+           "unexpected compressed klass shift!");
+    ld(xbase, ExternalAddress(CompressedKlassPointers::base_addr()));
+  } else if (CompressedKlassPointers::base() == nullptr) {
     if (CompressedKlassPointers::shift() != 0) {
       slli(dst, src, CompressedKlassPointers::shift());
     } else {
       mv(dst, src);
     }
     return;
+  } else {
+    mv(xbase, (uintptr_t)CompressedKlassPointers::base());
   }
 
-  Register xbase = tmp;
-
-  mv(xbase, (uintptr_t)CompressedKlassPointers::base());
-
   if (CompressedKlassPointers::shift() != 0) {
     // dst = (src << shift) + xbase
     shadd(dst, src, xbase, dst /* temporary, dst != xbase */, CompressedKlassPointers::shift());
@@ -3962,6 +3992,28 @@ void MacroAssembler::encode_klass_not_null(Register r, Register tmp) {
 }
 
 void MacroAssembler::encode_klass_not_null(Register dst, Register src, Register tmp) {
+  Register xbase = dst;
+  if (dst == src) {
+    xbase = tmp;
+  }
+
+  if (AOTCodeCache::is_on_for_dump()) {
+    // We are generating code during AOT buildup that will run in *future* processes
+    // with likely different encoding settings. Therefore, we have to load the
+    // encoding base dynamically and must not take the base-value dependent zext
+    // short cut below. Note that we only need to do this for base; the encoding
+    // shift is the same at build and run time: the standard precomputed shift.
+    assert(CompressedKlassPointers::shift() == ArchiveBuilder::precomputed_narrow_klass_shift(),
+           "unexpected compressed klass shift!");
+    assert_different_registers(src, xbase);
+    ld(xbase, ExternalAddress(CompressedKlassPointers::base_addr()));
+    sub(dst, src, xbase);
+    if (CompressedKlassPointers::shift() != 0) {
+      srli(dst, dst, CompressedKlassPointers::shift());
+    }
+    return;
+  }
+
   if (CompressedKlassPointers::base() == nullptr) {
     if (CompressedKlassPointers::shift() != 0) {
       srli(dst, src, CompressedKlassPointers::shift());
@@ -3977,11 +4029,6 @@ void MacroAssembler::encode_klass_not_null(Register dst, Register src, Register
     return;
   }
 
-  Register xbase = dst;
-  if (dst == src) {
-    xbase = tmp;
-  }
-
   assert_different_registers(src, xbase);
   mv(xbase, (uintptr_t)CompressedKlassPointers::base());
   sub(dst, src, xbase);
@@ -4018,11 +4065,18 @@ void  MacroAssembler::decode_heap_oop(Register d, Register s) {
       slli(d, s, CompressedOops::shift());
     }
   } else {
-    Label done;
-    mv(d, s);
-    beqz(s, done);
-    shadd(d, s, xheapbase, d, LogMinObjAlignmentInBytes);
-    bind(done);
+    assert(LogMinObjAlignmentInBytes == CompressedOops::shift(), "decode alg wrong");
+    if (UseZicond) {
+      assert_different_registers(s, t0);
+      shadd(t0, s, xheapbase, t0, LogMinObjAlignmentInBytes);
+      czero_eqz(d, t0, s);   // d = s == 0 ? 0 : t0
+    } else {
+      Label done;
+      mv(d, s);
+      beqz(s, done);
+      shadd(d, s, xheapbase, d, LogMinObjAlignmentInBytes);
+      bind(done);
+    }
   }
   verify_oop_msg(d, "broken oop in decode_heap_oop");
 }
@@ -5333,7 +5387,9 @@ void MacroAssembler::verify_secondary_supers_table(Register r_sub_klass,
     mv(x11, r_sub_klass);
     mv(x12, tmp3);
     mv(x13, result);
-    mv(x14, (address)("mismatch"));
+    const char* msg = "mismatch";
+    const char* str = (code_section()->scratch_emit()) ? msg : AOTCodeCache::add_C_string(msg);
+    la(x14, ExternalAddress((address) str));
     rt_call(CAST_FROM_FN_PTR(address, Klass::on_secondary_supers_verification_failure));
     should_not_reach_here();
   }
@@ -5363,8 +5419,7 @@ void MacroAssembler::get_thread(Register thread) {
                       RegSet::range(x28, x31) + ra - thread;
   push_reg(saved_regs, sp);
 
-  mv(t1, CAST_FROM_FN_PTR(address, Thread::current));
-  jalr(t1);
+  rt_call(CAST_FROM_FN_PTR(address, Thread::current), t1, t0);
   if (thread != c_rarg0) {
     mv(thread, c_rarg0);
   }
@@ -5374,12 +5429,33 @@ void MacroAssembler::get_thread(Register thread) {
 }
 
 void MacroAssembler::load_byte_map_base(Register reg) {
+#if INCLUDE_CDS
+  if (AOTCodeCache::is_on_for_dump()) {
+    address byte_map_base_adr = AOTRuntimeConstants::card_table_base_address();
+    ld(reg, ExternalAddress(byte_map_base_adr));
+    return;
+  }
+#endif
   CardTableBarrierSet* ctbs = CardTableBarrierSet::barrier_set();
   // Strictly speaking the card table base isn't an address at all, and it might
   // even be negative. It is thus materialised as a constant.
   mv(reg, (uint64_t)ctbs->card_table_base_const());
 }
 
+void MacroAssembler::load_aotrc_address(Register reg, address a) {
+#if INCLUDE_CDS
+  assert(AOTRuntimeConstants::contains(a), "address out of range for data area");
+  if (AOTCodeCache::is_on_for_dump()) {
+    // all aotrc field addresses should be registered in the AOTCodeCache address table
+    la(reg, ExternalAddress(a));
+  } else {
+    mv(reg, (intptr_t)a);
+  }
+#else
+  ShouldNotReachHere();
+#endif
+}
+
 void MacroAssembler::build_frame(int framesize) {
   assert(framesize >= 2, "framesize must include space for FP/RA");
   assert(framesize % (2*wordSize) == 0, "must preserve 2*wordSize alignment");
@@ -7035,10 +7111,8 @@ void MacroAssembler::fast_lock(Register basic_lock, Register obj, Register tmp1,
   // instruction emitted as it is part of C1's null check semantics.
   ld(mark, Address(obj, oopDesc::mark_offset_in_bytes()));
 
-  if (UseObjectMonitorTable) {
-    // Clear cache in case fast locking succeeds or we need to take the slow-path.
-    sd(zr, Address(basic_lock, BasicObjectLock::lock_offset() + in_ByteSize((BasicLock::object_monitor_cache_offset_in_bytes()))));
-  }
+  // Clear cache in case fast locking succeeds or we need to take the slow-path.
+  sd(zr, Address(basic_lock, BasicObjectLock::lock_offset() + in_ByteSize((BasicLock::object_monitor_cache_offset_in_bytes()))));
 
   if (DiagnoseSyncOnValueBasedClasses != 0) {
     load_klass(tmp1, obj);

src/hotspot/cpu/riscv/macroAssembler_riscv.hpp

@@ -28,6 +28,7 @@
 #define CPU_RISCV_MACROASSEMBLER_RISCV_HPP
 
 #include "asm/assembler.inline.hpp"
+#include "code/aotCodeCache.hpp"
 #include "code/vmreg.hpp"
 #include "metaprogramming/enableIf.hpp"
 #include "oops/compressedOops.hpp"
@@ -684,6 +685,9 @@ class MacroAssembler: public Assembler {
   void bltz(Register Rs, const address dest);
   void bgtz(Register Rs, const address dest);
 
+  void cmov_zicond_eqz(Register dst, Register src, Register cond, Register tmp = t0);
+  void cmov_zicond_nez(Register dst, Register src, Register cond, Register tmp = t0);
+
   void cmov_eq(Register cmp1, Register cmp2, Register dst, Register src);
   void cmov_ne(Register cmp1, Register cmp2, Register dst, Register src);
   void cmov_le(Register cmp1, Register cmp2, Register dst, Register src);
@@ -770,7 +774,7 @@ class MacroAssembler: public Assembler {
   // is used to keep the entry address for jalr/movptr.
   // Uses call() for intra code cache, else movptr + jalr.
   // Clobebrs t1
-  void rt_call(address dest, Register tmp = t1);
+  void rt_call(address dest, Register tmp1 = t1, Register tmp2 = noreg);
 
   // ret: jalr x0, 0(x1)
   inline void ret() {
@@ -1291,6 +1295,9 @@ public:
 
   void load_byte_map_base(Register reg);
 
+  // Load a constant address in the AOT Runtime Constants area
+  void load_aotrc_address(Register reg, address a);
+
   void bang_stack_with_offset(int offset) {
     // stack grows down, caller passes positive offset
     assert(offset > 0, "must bang with negative offset");
@@ -1835,7 +1842,7 @@ public:
   static bool is_pc_relative_at(address branch);
 
   static bool is_membar(address addr) {
-    return (Bytes::get_native_u4(addr) & 0x7f) == 0b1111 && extract_funct3(addr) == 0;
+    return (Assembler::ld_instr(addr) & 0x7f) == 0b1111 && extract_funct3(addr) == 0;
   }
   static uint32_t get_membar_kind(address addr);
   static void set_membar_kind(address addr, uint32_t order_kind);

src/hotspot/cpu/riscv/nativeInst_riscv.cpp

@@ -234,7 +234,7 @@ void NativeMovConstReg::verify() {
 intptr_t NativeMovConstReg::data() const {
   address addr = MacroAssembler::target_addr_for_insn(instruction_address());
   if (maybe_cpool_ref(instruction_address())) {
-    return Bytes::get_native_u8(addr);
+    return MacroAssembler::get_native_u8(addr);
   } else {
     return (intptr_t)addr;
   }
@@ -243,7 +243,7 @@ intptr_t NativeMovConstReg::data() const {
 void NativeMovConstReg::set_data(intptr_t x) {
   if (maybe_cpool_ref(instruction_address())) {
     address addr = MacroAssembler::target_addr_for_insn(instruction_address());
-    Bytes::put_native_u8(addr, x);
+    MacroAssembler::put_native_u8(addr, x);
   } else {
     // Store x into the instruction stream.
     MacroAssembler::pd_patch_instruction_size(instruction_address(), (address)x);
@@ -259,11 +259,11 @@ void NativeMovConstReg::set_data(intptr_t x) {
     while (iter.next()) {
       if (iter.type() == relocInfo::oop_type) {
         oop* oop_addr = iter.oop_reloc()->oop_addr();
-        Bytes::put_native_u8((address)oop_addr, x);
+        MacroAssembler::put_native_u8((address)oop_addr, x);
         break;
       } else if (iter.type() == relocInfo::metadata_type) {
         Metadata** metadata_addr = iter.metadata_reloc()->metadata_addr();
-        Bytes::put_native_u8((address)metadata_addr, x);
+        MacroAssembler::put_native_u8((address)metadata_addr, x);
         break;
       }
     }

src/hotspot/cpu/riscv/nativeInst_riscv.hpp

@@ -78,19 +78,19 @@ class NativeInstruction {
 
  protected:
   address addr_at(int offset) const { return address(this) + offset; }
-  jint     int_at(int offset) const { return (jint)      Bytes::get_native_u4(addr_at(offset));  }
-  juint   uint_at(int offset) const { return             Bytes::get_native_u4(addr_at(offset));  }
-  address  ptr_at(int offset) const { return (address)   Bytes::get_native_u8(addr_at(offset));  }
-  oop      oop_at(int offset) const { return cast_to_oop(Bytes::get_native_u8(addr_at(offset))); }
+  jint     int_at(int offset) const { return (jint)      MacroAssembler::get_native_u4(addr_at(offset));  }
+  juint   uint_at(int offset) const { return             MacroAssembler::get_native_u4(addr_at(offset));  }
+  address  ptr_at(int offset) const { return (address)   MacroAssembler::get_native_u8(addr_at(offset));  }
+  oop      oop_at(int offset) const { return cast_to_oop(MacroAssembler::get_native_u8(addr_at(offset))); }
 
 
-  void  set_int_at(int offset, jint i)      { Bytes::put_native_u4(addr_at(offset), i); }
-  void set_uint_at(int offset, jint i)      { Bytes::put_native_u4(addr_at(offset), i); }
-  void  set_ptr_at(int offset, address ptr) { Bytes::put_native_u8(addr_at(offset), (u8)ptr); }
-  void  set_oop_at(int offset, oop o)       { Bytes::put_native_u8(addr_at(offset), cast_from_oop<u8>(o)); }
+  void  set_int_at(int offset, jint i)      { MacroAssembler::put_native_u4(addr_at(offset), i); }
+  void set_uint_at(int offset, juint i)     { MacroAssembler::put_native_u4(addr_at(offset), i); }
+  void  set_ptr_at(int offset, address ptr) { MacroAssembler::put_native_u8(addr_at(offset), (u8)ptr); }
+  void  set_oop_at(int offset, oop o)       { MacroAssembler::put_native_u8(addr_at(offset), cast_from_oop<u8>(o)); }
 
-  static void     set_data64_at(address dest, uint64_t data) { Bytes::put_native_u8(dest, (u8)data); }
-  static uint64_t get_data64_at(address src)                 { return Bytes::get_native_u8(src); }
+  static void     set_data64_at(address dest, uint64_t data) { MacroAssembler::put_native_u8(dest, (u8)data); }
+  static uint64_t get_data64_at(address src)                 { return MacroAssembler::get_native_u8(src); }
 
  public:
   inline friend NativeInstruction* nativeInstruction_at(address addr);

src/hotspot/cpu/riscv/relocInfo_riscv.cpp

@@ -44,7 +44,7 @@ void Relocation::pd_set_data_value(address x, bool verify_only) {
       if (MacroAssembler::is_load_pc_relative_at(addr())) {
         address constptr = (address)code()->oop_addr_at(reloc->oop_index());
         bytes = MacroAssembler::pd_patch_instruction_size(addr(), constptr);
-        assert((address)Bytes::get_native_u8(constptr) == x, "error in oop relocation");
+        assert((address)MacroAssembler::get_native_u8(constptr) == x, "error in oop relocation");
       } else {
         bytes = MacroAssembler::patch_oop(addr(), x);
       }

src/hotspot/cpu/riscv/riscv.ad

@@ -1553,9 +1553,34 @@ uint MachSpillCopyNode::implementation(C2_MacroAssembler *masm, PhaseRegAlloc *r
   int src_offset = ra_->reg2offset(src_lo);
   int dst_offset = ra_->reg2offset(dst_lo);
 
-  if (bottom_type()->isa_vect() != nullptr) {
-    uint ireg = ideal_reg();
-    if (ireg == Op_VecA && masm) {
+  // Stack-to-stack copies use t0 for the value. Bail out if a destination
+  // address also needs t0 to materialize an offset outside the 12-bit range.
+  if (src_lo_rc == rc_stack && dst_lo_rc == rc_stack) {
+    int last_dst_offset = dst_offset;
+    if (bottom_type()->isa_vect() != nullptr) {
+      if (!bottom_type()->isa_pvectmask()) {
+        assert(ideal_reg() == Op_VecA, "Must be Op_VecA");
+        int vector_reg_size_in_bytes = Matcher::scalable_vector_reg_size(T_BYTE);
+        last_dst_offset += vector_reg_size_in_bytes - 8;
+      } else {
+        assert(ideal_reg() == Op_RegVectMask, "Must be Op_RegVectMask");
+        int vmask_size_in_bytes = Matcher::scalable_predicate_reg_slots() * 32 / 8;
+        last_dst_offset += vmask_size_in_bytes - 4;
+      }
+    }
+
+    if (masm != nullptr && !Assembler::is_simm12(last_dst_offset)) {
+      // size() emits into a scratch buffer where recording a failure is not allowed.
+      if (!C->output()->in_scratch_emit_size()) {
+        C->record_method_not_compilable("unsupported large stack-to-stack spill copy");
+      }
+      return 0;
+    }
+  }
+
+  if (bottom_type()->isa_vect() != nullptr && masm != nullptr) {
+    if (!bottom_type()->isa_pvectmask()) {
+      assert(ideal_reg() == Op_VecA, "Must be Op_VecA");
       int vector_reg_size_in_bytes = Matcher::scalable_vector_reg_size(T_BYTE);
       if (src_lo_rc == rc_stack && dst_lo_rc == rc_stack) {
         // stack to stack
@@ -1574,7 +1599,8 @@ uint MachSpillCopyNode::implementation(C2_MacroAssembler *masm, PhaseRegAlloc *r
       } else {
         ShouldNotReachHere();
       }
-    } else if (bottom_type()->isa_pvectmask() && masm) {
+    } else {
+      assert(ideal_reg() == Op_RegVectMask, "Must be Op_RegVectMask");
       int vmask_size_in_bytes = Matcher::scalable_predicate_reg_slots() * 32 / 8;
       if (src_lo_rc == rc_stack && dst_lo_rc == rc_stack) {
         // stack to stack
@@ -1677,14 +1703,11 @@ uint MachSpillCopyNode::implementation(C2_MacroAssembler *masm, PhaseRegAlloc *r
       st->print("%s", Matcher::regName[dst_lo]);
     }
     if (bottom_type()->isa_vect() && !bottom_type()->isa_pvectmask()) {
-      int vsize = 0;
-      if (ideal_reg() == Op_VecA) {
-        vsize = Matcher::scalable_vector_reg_size(T_BYTE) * 8;
-      } else {
-        ShouldNotReachHere();
-      }
+      assert(ideal_reg() == Op_VecA, "Must be Op_VecA");
+      int vsize = Matcher::scalable_vector_reg_size(T_BYTE) * 8;
       st->print("\t# vector spill size = %d", vsize);
-    } else if (ideal_reg() == Op_RegVectMask) {
+    } else if (bottom_type()->isa_pvectmask()) {
+      assert(ideal_reg() == Op_RegVectMask, "Must be Op_RegVectMask");
       assert(Matcher::supports_scalable_vector(), "bad register type for spill");
       int vsize = Matcher::scalable_predicate_reg_slots() * 32;
       st->print("\t# vmask spill size = %d", vsize);
@@ -2400,10 +2423,7 @@ encode %{
       // Make the anchor frame walkable
       __ la(t0, retaddr);
       __ sd(t0, Address(xthread, JavaThread::last_Java_pc_offset()));
-      int32_t offset = 0;
-      // No relocation needed
-      __ movptr(t1, entry, offset, t0); // lui + lui + slli + add
-      __ jalr(t1, offset);
+      __ rt_call(entry, t1, t0);
       __ bind(retaddr);
       __ post_call_nop();
     }
@@ -2800,6 +2820,18 @@ operand immP_1()
   interface(CONST_INTER);
 %}
 
+// AOT Runtime Constants Address
+operand immAOTRuntimeConstantsAddress()
+%{
+  // Check if the address is in the range of AOT Runtime Constants
+  predicate(AOTRuntimeConstants::contains((address)(n->get_ptr())));
+  match(ConP);
+
+  op_cost(0);
+  format %{ %}
+  interface(CONST_INTER);
+%}
+
 // Int Immediate: low 16-bit mask
 operand immI_16bits()
 %{
@@ -2818,6 +2850,16 @@ operand immIpowerOf2() %{
   interface(CONST_INTER);
 %}
 
+// Long Immediate: low 16-bit mask
+operand immL_16bits()
+%{
+  predicate(n->get_long() == 0xFFFFL);
+  match(ConL);
+  op_cost(0);
+  format %{ %}
+  interface(CONST_INTER);
+%}
+
 // Long Immediate: low 32-bit mask
 operand immL_32bits()
 %{
@@ -4761,6 +4803,20 @@ instruct loadConP1(iRegPNoSp dst, immP_1 con)
   ins_pipe(ialu_imm);
 %}
 
+instruct loadAOTRCAddress(iRegPNoSp dst, immAOTRuntimeConstantsAddress con)
+%{
+  match(Set dst con);
+
+  ins_cost(ALU_COST);
+  format %{ "la  $dst, $con\t# aotrc, #@loadAOTRCAddress" %}
+
+  ins_encode %{
+    __ load_aotrc_address($dst$$Register, (address)$con$$constant);
+  %}
+
+  ins_pipe(ialu_imm);
+%}
+
 // Load Narrow Pointer Constant
 instruct loadConN(iRegNNoSp dst, immN con)
 %{
@@ -8531,6 +8587,157 @@ instruct convI2L_reg_reg(iRegLNoSp dst, iRegIorL2I src)
   ins_pipe(ialu_reg);
 %}
 
+// Fused int-producer + ConvI2L rules.
+instruct convI2L_addI_reg_reg(iRegLNoSp dst, iRegIorL2I src1, iRegIorL2I src2) %{
+  match(Set dst (ConvI2L (AddI src1 src2)));
+
+  ins_cost(ALU_COST);
+  format %{ "addw  $dst, $src1, $src2\t#@convI2L_addI_reg_reg" %}
+
+  ins_encode %{
+    __ addw(as_Register($dst$$reg),
+            as_Register($src1$$reg),
+            as_Register($src2$$reg));
+  %}
+
+  ins_pipe(ialu_reg_reg);
+%}
+
+instruct convI2L_addI_reg_imm(iRegLNoSp dst, iRegIorL2I src1, immIAdd src2) %{
+  match(Set dst (ConvI2L (AddI src1 src2)));
+
+  ins_cost(ALU_COST);
+  format %{ "addiw  $dst, $src1, $src2\t#@convI2L_addI_reg_imm" %}
+
+  ins_encode %{
+    __ addiw(as_Register($dst$$reg),
+             as_Register($src1$$reg),
+             $src2$$constant);
+  %}
+
+  ins_pipe(ialu_reg_imm);
+%}
+
+instruct convI2L_subI_reg_reg(iRegLNoSp dst, iRegIorL2I src1, iRegIorL2I src2) %{
+  match(Set dst (ConvI2L (SubI src1 src2)));
+
+  ins_cost(ALU_COST);
+  format %{ "subw  $dst, $src1, $src2\t#@convI2L_subI_reg_reg" %}
+
+  ins_encode %{
+    __ subw(as_Register($dst$$reg),
+            as_Register($src1$$reg),
+            as_Register($src2$$reg));
+  %}
+
+  ins_pipe(ialu_reg_reg);
+%}
+
+instruct convI2L_mulI_reg_reg(iRegLNoSp dst, iRegIorL2I src1, iRegIorL2I src2) %{
+  match(Set dst (ConvI2L (MulI src1 src2)));
+
+  ins_cost(IMUL_COST);
+  format %{ "mulw  $dst, $src1, $src2\t#@convI2L_mulI_reg_reg" %}
+
+  ins_encode %{
+    __ mulw(as_Register($dst$$reg),
+            as_Register($src1$$reg),
+            as_Register($src2$$reg));
+  %}
+
+  ins_pipe(imul_reg_reg);
+%}
+
+instruct convI2L_lShiftI_reg_reg(iRegLNoSp dst, iRegIorL2I src1, iRegIorL2I src2) %{
+  match(Set dst (ConvI2L (LShiftI src1 src2)));
+
+  ins_cost(ALU_COST);
+  format %{ "sllw  $dst, $src1, $src2\t#@convI2L_lShiftI_reg_reg" %}
+
+  ins_encode %{
+    __ sllw(as_Register($dst$$reg),
+            as_Register($src1$$reg),
+            as_Register($src2$$reg));
+  %}
+
+  ins_pipe(ialu_reg_reg_vshift);
+%}
+
+instruct convI2L_lShiftI_reg_imm(iRegLNoSp dst, iRegIorL2I src1, immI src2) %{
+  match(Set dst (ConvI2L (LShiftI src1 src2)));
+
+  ins_cost(ALU_COST);
+  format %{ "slliw  $dst, $src1, ($src2 & 0x1f)\t#@convI2L_lShiftI_reg_imm" %}
+
+  ins_encode %{
+    __ slliw(as_Register($dst$$reg),
+             as_Register($src1$$reg),
+             (unsigned) $src2$$constant & 0x1f);
+  %}
+
+  ins_pipe(ialu_reg_shift);
+%}
+
+instruct convI2L_urShiftI_reg_reg(iRegLNoSp dst, iRegIorL2I src1, iRegIorL2I src2) %{
+  match(Set dst (ConvI2L (URShiftI src1 src2)));
+
+  ins_cost(ALU_COST);
+  format %{ "srlw  $dst, $src1, $src2\t#@convI2L_urShiftI_reg_reg" %}
+
+  ins_encode %{
+    __ srlw(as_Register($dst$$reg),
+            as_Register($src1$$reg),
+            as_Register($src2$$reg));
+  %}
+
+  ins_pipe(ialu_reg_reg_vshift);
+%}
+
+instruct convI2L_urShiftI_reg_imm(iRegLNoSp dst, iRegIorL2I src1, immI src2) %{
+  match(Set dst (ConvI2L (URShiftI src1 src2)));
+
+  ins_cost(ALU_COST);
+  format %{ "srliw  $dst, $src1, ($src2 & 0x1f)\t#@convI2L_urShiftI_reg_imm" %}
+
+  ins_encode %{
+    __ srliw(as_Register($dst$$reg),
+             as_Register($src1$$reg),
+             (unsigned) $src2$$constant & 0x1f);
+  %}
+
+  ins_pipe(ialu_reg_shift);
+%}
+
+instruct convI2L_rShiftI_reg_reg(iRegLNoSp dst, iRegIorL2I src1, iRegIorL2I src2) %{
+  match(Set dst (ConvI2L (RShiftI src1 src2)));
+
+  ins_cost(ALU_COST);
+  format %{ "sraw  $dst, $src1, $src2\t#@convI2L_rShiftI_reg_reg" %}
+
+  ins_encode %{
+    __ sraw(as_Register($dst$$reg),
+            as_Register($src1$$reg),
+            as_Register($src2$$reg));
+  %}
+
+  ins_pipe(ialu_reg_reg_vshift);
+%}
+
+instruct convI2L_rShiftI_reg_imm(iRegLNoSp dst, iRegIorL2I src1, immI src2) %{
+  match(Set dst (ConvI2L (RShiftI src1 src2)));
+
+  ins_cost(ALU_COST);
+  format %{ "sraiw  $dst, $src1, ($src2 & 0x1f)\t#@convI2L_rShiftI_reg_imm" %}
+
+  ins_encode %{
+    __ sraiw(as_Register($dst$$reg),
+             as_Register($src1$$reg),
+             (unsigned) $src2$$constant & 0x1f);
+  %}
+
+  ins_pipe(ialu_reg_shift);
+%}
+
 instruct convL2I_reg(iRegINoSp dst, iRegL src) %{
   match(Set dst (ConvL2I src));
 

src/hotspot/cpu/riscv/riscv_b.ad

@@ -183,7 +183,37 @@ instruct convI2UL_reg_reg_b(iRegLNoSp dst, iRegIorL2I src, immL_32bits mask) %{
     __ zext_w(as_Register($dst$$reg), as_Register($src$$reg));
   %}
 
-  ins_pipe(ialu_reg_shift);
+  ins_pipe(ialu_reg);
+%}
+
+// And with a low 16-bit mask
+instruct andL_16bits_b(iRegLNoSp dst, iRegL src, immL_16bits mask) %{
+  predicate(UseZbb);
+  match(Set dst (AndL src mask));
+
+  format %{ "zext.h  $dst, $src\t#@andL_16bits_b" %}
+
+  ins_cost(ALU_COST);
+  ins_encode %{
+    __ zext_h(as_Register($dst$$reg), as_Register($src$$reg));
+  %}
+
+  ins_pipe(ialu_reg);
+%}
+
+// And with a low 32-bit mask
+instruct andL_32bits_b(iRegLNoSp dst, iRegL src, immL_32bits mask) %{
+  predicate(UseZba);
+  match(Set dst (AndL src mask));
+
+  format %{ "zext.w  $dst, $src\t#@andL_32bits_b" %}
+
+  ins_cost(ALU_COST);
+  ins_encode %{
+    __ zext_w(as_Register($dst$$reg), as_Register($src$$reg));
+  %}
+
+  ins_pipe(ialu_reg);
 %}
 
 // BSWAP instructions

src/hotspot/cpu/riscv/riscv_v.ad

@@ -41,7 +41,7 @@ source %{
       __ vsex_v(reg, base, sew, vm);
     } else {
       if (vm == Assembler::v0_t) {
-        __ vxor_vv(reg, reg, reg);
+        __ vmv_v_i(reg, 0);
       }
       __ vlex_v(reg, base, sew, vm);
     }
@@ -241,7 +241,7 @@ instruct vstoremask(vReg dst, vRegMask_V0 v0, immI size) %{
   format %{ "vstoremask $dst, V0 # elem size is $size byte[s]" %}
   ins_encode %{
     __ vsetvli_helper(T_BOOLEAN, Matcher::vector_length(this));
-    __ vxor_vv(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg));
+    __ vmv_v_i(as_VectorRegister($dst$$reg), 0);
     __ vmerge_vim(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg), 1);
   %}
   ins_pipe(pipe_slow);
@@ -1580,6 +1580,42 @@ instruct vnotL_masked(vReg dst_src, immI_M1 m1, vRegMask_V0 v0) %{
   ins_pipe(pipe_slow);
 %}
 
+// vector integer div
+
+instruct vdiv(vReg dst, vReg src1, vReg src2) %{
+  match(Set dst (DivVB src1 src2));
+  match(Set dst (DivVS src1 src2));
+  match(Set dst (DivVI src1 src2));
+  match(Set dst (DivVL src1 src2));
+  format %{ "vdiv $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vdiv_vv(as_VectorRegister($dst$$reg),
+               as_VectorRegister($src1$$reg),
+               as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+// vector integer div - predicated
+
+instruct vdiv_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{
+  match(Set dst_src1 (DivVB (Binary dst_src1 src2) v0));
+  match(Set dst_src1 (DivVS (Binary dst_src1 src2) v0));
+  match(Set dst_src1 (DivVI (Binary dst_src1 src2) v0));
+  match(Set dst_src1 (DivVL (Binary dst_src1 src2) v0));
+  format %{ "vdiv_masked $dst_src1, $dst_src1, $src2, $v0" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vdiv_vv(as_VectorRegister($dst_src1$$reg),
+               as_VectorRegister($dst_src1$$reg),
+               as_VectorRegister($src2$$reg), Assembler::v0_t);
+  %}
+  ins_pipe(pipe_slow);
+%}
+
 // vector float div
 
 instruct vdiv_hfp(vReg dst, vReg src1, vReg src2) %{
@@ -4482,7 +4518,7 @@ instruct vmaskAllL(vRegMask dst, iRegL src) %{
 
 // ------------------------------ Vector mask basic OPs ------------------------
 
-// vector mask logical ops: and/or/xor
+// vector mask logical ops
 
 instruct vmask_and(vRegMask dst, vRegMask src1, vRegMask src2) %{
   match(Set dst (AndVMask src1 src2));
@@ -4497,6 +4533,162 @@ instruct vmask_and(vRegMask dst, vRegMask src1, vRegMask src2) %{
   ins_pipe(pipe_slow);
 %}
 
+instruct vmask_and_notI(vRegMask dst, vRegMask src1, vRegMask src2, immI_M1 m1) %{
+  match(Set dst (AndVMask src1 (XorVMask src2 (MaskAll m1))));
+  format %{ "vmask_and_notI $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmandn_mm(as_VectorRegister($dst$$reg),
+                 as_VectorRegister($src1$$reg),
+                 as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_and_notL(vRegMask dst, vRegMask src1, vRegMask src2, immL_M1 m1) %{
+  match(Set dst (AndVMask src1 (XorVMask src2 (MaskAll m1))));
+  format %{ "vmask_and_notL $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmandn_mm(as_VectorRegister($dst$$reg),
+                 as_VectorRegister($src1$$reg),
+                 as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_or_notI(vRegMask dst, vRegMask src1, vRegMask src2, immI_M1 m1) %{
+  match(Set dst (OrVMask src1 (XorVMask src2 (MaskAll m1))));
+  format %{ "vmask_or_notI $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmorn_mm(as_VectorRegister($dst$$reg),
+                as_VectorRegister($src1$$reg),
+                as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_or_notL(vRegMask dst, vRegMask src1, vRegMask src2, immL_M1 m1) %{
+  match(Set dst (OrVMask src1 (XorVMask src2 (MaskAll m1))));
+  format %{ "vmask_or_notL $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmorn_mm(as_VectorRegister($dst$$reg),
+                as_VectorRegister($src1$$reg),
+                as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_nandI(vRegMask dst, vRegMask src1, vRegMask src2, immI_M1 m1) %{
+  match(Set dst (XorVMask (AndVMask src1 src2) (MaskAll m1)));
+  format %{ "vmask_nandI $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmnand_mm(as_VectorRegister($dst$$reg),
+                 as_VectorRegister($src1$$reg),
+                 as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_nandL(vRegMask dst, vRegMask src1, vRegMask src2, immL_M1 m1) %{
+  match(Set dst (XorVMask (AndVMask src1 src2) (MaskAll m1)));
+  format %{ "vmask_nandL $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmnand_mm(as_VectorRegister($dst$$reg),
+                 as_VectorRegister($src1$$reg),
+                 as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_norI(vRegMask dst, vRegMask src1, vRegMask src2, immI_M1 m1) %{
+  match(Set dst (XorVMask (OrVMask src1 src2) (MaskAll m1)));
+  format %{ "vmask_norI $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmnor_mm(as_VectorRegister($dst$$reg),
+                as_VectorRegister($src1$$reg),
+                as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_norL(vRegMask dst, vRegMask src1, vRegMask src2, immL_M1 m1) %{
+  match(Set dst (XorVMask (OrVMask src1 src2) (MaskAll m1)));
+  format %{ "vmask_norL $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmnor_mm(as_VectorRegister($dst$$reg),
+                as_VectorRegister($src1$$reg),
+                as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_xnorI(vRegMask dst, vRegMask src1, vRegMask src2, immI_M1 m1) %{
+  match(Set dst (XorVMask (XorVMask src1 src2) (MaskAll m1)));
+  match(Set dst (XorVMask src1 (XorVMask src2 (MaskAll m1))));
+  format %{ "vmask_xnorI $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmxnor_mm(as_VectorRegister($dst$$reg),
+                 as_VectorRegister($src1$$reg),
+                 as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_xnorL(vRegMask dst, vRegMask src1, vRegMask src2, immL_M1 m1) %{
+  match(Set dst (XorVMask (XorVMask src1 src2) (MaskAll m1)));
+  match(Set dst (XorVMask src1 (XorVMask src2 (MaskAll m1))));
+  format %{ "vmask_xnorL $dst, $src1, $src2" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmxnor_mm(as_VectorRegister($dst$$reg),
+                 as_VectorRegister($src1$$reg),
+                 as_VectorRegister($src2$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_notI(vRegMask dst, vRegMask src, immI_M1 m1) %{
+  match(Set dst (XorVMask src (MaskAll m1)));
+  format %{ "vmask_notI $dst, $src" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmnot_m(as_VectorRegister($dst$$reg),
+               as_VectorRegister($src$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
+instruct vmask_notL(vRegMask dst, vRegMask src, immL_M1 m1) %{
+  match(Set dst (XorVMask src (MaskAll m1)));
+  format %{ "vmask_notL $dst, $src" %}
+  ins_encode %{
+    BasicType bt = Matcher::vector_element_basic_type(this);
+    __ vsetvli_helper(bt, Matcher::vector_length(this));
+    __ vmnot_m(as_VectorRegister($dst$$reg),
+               as_VectorRegister($src$$reg));
+  %}
+  ins_pipe(pipe_slow);
+%}
+
 instruct vmask_or(vRegMask dst, vRegMask src1, vRegMask src2) %{
   match(Set dst (OrVMask src1 src2));
   format %{ "vmask_or $dst, $src1, $src2" %}
@@ -4809,7 +5001,7 @@ instruct vcvtFtoL(vReg dst, vReg src, vRegMask_V0 v0) %{
   format %{ "vcvtFtoL $dst, $src" %}
   ins_encode %{
     __ vsetvli_helper(T_LONG, Matcher::vector_length(this));
-    __ vxor_vv(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg));
+    __ vmv_v_i(as_VectorRegister($dst$$reg), 0);
     __ vsetvli_helper(T_FLOAT, Matcher::vector_length(this), Assembler::mf2);
     __ vmfeq_vv(as_VectorRegister($v0$$reg), as_VectorRegister($src$$reg), as_VectorRegister($src$$reg));
     __ vfwcvt_rtz_x_f_v(as_VectorRegister($dst$$reg), as_VectorRegister($src$$reg), Assembler::v0_t);
@@ -4842,7 +5034,7 @@ instruct vcvtDtoX_narrow(vReg dst, vReg src, vRegMask_V0 v0) %{
     __ vsetvli_helper(T_DOUBLE, Matcher::vector_length(this));
     __ vmfeq_vv(as_VectorRegister($v0$$reg), as_VectorRegister($src$$reg), as_VectorRegister($src$$reg));
     __ vsetvli_helper(T_INT, Matcher::vector_length(this), Assembler::mf2);
-    __ vxor_vv(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg));
+    __ vmv_v_i(as_VectorRegister($dst$$reg), 0);
     __ vfncvt_rtz_x_f_w(as_VectorRegister($dst$$reg), as_VectorRegister($src$$reg), Assembler::v0_t);
     BasicType bt = Matcher::vector_element_basic_type(this);
     if (bt == T_BYTE || bt == T_SHORT) {
@@ -4903,7 +5095,7 @@ instruct reinterpretResize(vReg dst, vReg src) %{
            "invalid vector length");
     BasicType bt = Matcher::vector_element_basic_type(this);
     __ vsetvli_helper(bt, Matcher::vector_length(this));
-    __ vxor_vv(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg));
+    __ vmv_v_i(as_VectorRegister($dst$$reg), 0);
     __ vsetvli_helper(T_BYTE, length_in_bytes_resize);
     __ vmv_v_v(as_VectorRegister($dst$$reg), as_VectorRegister($src$$reg));
   %}
@@ -4931,7 +5123,7 @@ instruct vmask_reinterpret_diff_esize(vRegMask dst, vRegMask_V0 src, vReg tmp) %
   ins_encode %{
     BasicType from_bt = Matcher::vector_element_basic_type(this, $src);
     __ vsetvli_helper(from_bt, Matcher::vector_length(this, $src));
-    __ vxor_vv(as_VectorRegister($tmp$$reg), as_VectorRegister($tmp$$reg), as_VectorRegister($tmp$$reg));
+    __ vmv_v_i(as_VectorRegister($tmp$$reg), 0);
     __ vmerge_vim(as_VectorRegister($tmp$$reg), as_VectorRegister($tmp$$reg), -1);
     BasicType to_bt = Matcher::vector_element_basic_type(this);
     __ vsetvli_helper(to_bt, Matcher::vector_length(this));
@@ -4990,8 +5182,7 @@ instruct rearrange_masked(vReg dst, vReg src, vReg shuffle, vRegMask_V0 v0) %{
   ins_encode %{
     BasicType bt = Matcher::vector_element_basic_type(this);
     __ vsetvli_helper(bt, Matcher::vector_length(this));
-    __ vxor_vv(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg),
-               as_VectorRegister($dst$$reg));
+    __ vmv_v_i(as_VectorRegister($dst$$reg), 0);
     __ vrgather_vv(as_VectorRegister($dst$$reg), as_VectorRegister($src$$reg),
                    as_VectorRegister($shuffle$$reg), Assembler::v0_t);
   %}
@@ -5105,8 +5296,7 @@ instruct vcompress(vReg dst, vReg src, vRegMask_V0 v0) %{
   ins_encode %{
     BasicType bt = Matcher::vector_element_basic_type(this);
     __ vsetvli_helper(bt, Matcher::vector_length(this));
-    __ vxor_vv(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg),
-               as_VectorRegister($dst$$reg));
+    __ vmv_v_i(as_VectorRegister($dst$$reg), 0);
     __ vcompress_vm(as_VectorRegister($dst$$reg), as_VectorRegister($src$$reg),
                     as_VectorRegister($v0$$reg));
   %}
@@ -5121,8 +5311,7 @@ instruct vexpand(vReg dst, vReg src, vRegMask_V0 v0, vReg tmp) %{
     BasicType bt = Matcher::vector_element_basic_type(this);
     __ vsetvli_helper(bt, Matcher::vector_length(this));
     __ viota_m(as_VectorRegister($tmp$$reg), as_VectorRegister($v0$$reg));
-    __ vxor_vv(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg),
-               as_VectorRegister($dst$$reg));
+    __ vmv_v_i(as_VectorRegister($dst$$reg), 0);
     __ vrgather_vv(as_VectorRegister($dst$$reg), as_VectorRegister($src$$reg),
                    as_VectorRegister($tmp$$reg), Assembler::v0_t);
   %}
@@ -5383,8 +5572,7 @@ instruct gather_loadS_masked(vReg dst, indirect mem, vReg idx, vRegMask_V0 v0, v
     Assembler::SEW sew = Assembler::elemtype_to_sew(bt);
     __ vsetvli_helper(bt, Matcher::vector_length(this));
     __ vsll_vi(as_VectorRegister($tmp$$reg), as_VectorRegister($idx$$reg), (int)sew);
-    __ vxor_vv(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg),
-               as_VectorRegister($dst$$reg));
+    __ vmv_v_i(as_VectorRegister($dst$$reg), 0);
     __ vluxei32_v(as_VectorRegister($dst$$reg), as_Register($mem$$base),
                   as_VectorRegister($tmp$$reg), Assembler::v0_t);
  %}
@@ -5402,8 +5590,7 @@ instruct gather_loadD_masked(vReg dst, indirect mem, vReg idx, vRegMask_V0 v0, v
     __ vsetvli_helper(bt, Matcher::vector_length(this));
     __ vzext_vf2(as_VectorRegister($tmp$$reg), as_VectorRegister($idx$$reg));
     __ vsll_vi(as_VectorRegister($tmp$$reg), as_VectorRegister($tmp$$reg), (int)sew);
-    __ vxor_vv(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg),
-               as_VectorRegister($dst$$reg));
+    __ vmv_v_i(as_VectorRegister($dst$$reg), 0);
     __ vluxei64_v(as_VectorRegister($dst$$reg), as_Register($mem$$base),
                   as_VectorRegister($tmp$$reg), Assembler::v0_t);
  %}

src/hotspot/cpu/riscv/runtime_riscv.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2024, 2025, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2024, Red Hat Inc. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
@@ -26,6 +26,7 @@
 #ifdef COMPILER2
 #include "asm/macroAssembler.hpp"
 #include "asm/macroAssembler.inline.hpp"
+#include "code/aotCodeCache.hpp"
 #include "code/vmreg.hpp"
 #include "interpreter/interpreter.hpp"
 #include "opto/runtime.hpp"
@@ -58,10 +59,15 @@ public:
 
 //------------------------------generate_uncommon_trap_blob--------------------
 UncommonTrapBlob* OptoRuntime::generate_uncommon_trap_blob() {
+  const char* name = OptoRuntime::stub_name(StubId::c2_uncommon_trap_id);
+  CodeBlob* blob = AOTCodeCache::load_code_blob(AOTCodeEntry::C2Blob, BlobId::c2_uncommon_trap_id);
+  if (blob != nullptr) {
+    return blob->as_uncommon_trap_blob();
+  }
+
   // Allocate space for the code
   ResourceMark rm;
   // Setup code generation tools
-  const char* name = OptoRuntime::stub_name(StubId::c2_uncommon_trap_id);
   CodeBuffer buffer(name, 2048, 1024);
   if (buffer.blob() == nullptr) {
     return nullptr;
@@ -240,11 +246,12 @@ UncommonTrapBlob* OptoRuntime::generate_uncommon_trap_blob() {
   // Jump to interpreter
   __ ret();
 
-  // Make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
-  return UncommonTrapBlob::create(&buffer, oop_maps,
-                                                  SimpleRuntimeFrame::framesize >> 1);
+  UncommonTrapBlob* ut_blob = UncommonTrapBlob::create(&buffer, oop_maps,
+                                                       SimpleRuntimeFrame::framesize >> 1);
+  AOTCodeCache::store_code_blob(*ut_blob, AOTCodeEntry::C2Blob, BlobId::c2_uncommon_trap_id);
+  return ut_blob;
 }
 
 //------------------------------generate_exception_blob---------------------------
@@ -278,10 +285,15 @@ ExceptionBlob* OptoRuntime::generate_exception_blob() {
 
   assert(SimpleRuntimeFrame::framesize % 4 == 0, "sp not 16-byte aligned");
 
+  const char* name = OptoRuntime::stub_name(StubId::c2_exception_id);
+  CodeBlob* blob = AOTCodeCache::load_code_blob(AOTCodeEntry::C2Blob, BlobId::c2_exception_id);
+  if (blob != nullptr) {
+    return blob->as_exception_blob();
+  }
+
   // Allocate space for the code
   ResourceMark rm;
   // Setup code generation tools
-  const char* name = OptoRuntime::stub_name(StubId::c2_exception_id);
   CodeBuffer buffer(name, 2048, 1024);
   if (buffer.blob() == nullptr) {
     return nullptr;
@@ -376,10 +388,11 @@ ExceptionBlob* OptoRuntime::generate_exception_blob() {
 
   __ jr(t1);
 
-  // Make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   // Set exception blob
-  return ExceptionBlob::create(&buffer, oop_maps, SimpleRuntimeFrame::framesize >> 1);
+  ExceptionBlob* ex_blob = ExceptionBlob::create(&buffer, oop_maps, SimpleRuntimeFrame::framesize >> 1);
+  AOTCodeCache::store_code_blob(*ex_blob, AOTCodeEntry::C2Blob, BlobId::c2_exception_id);
+  return ex_blob;
 }
 #endif // COMPILER2

src/hotspot/cpu/riscv/sharedRuntime_riscv.cpp

@@ -27,6 +27,7 @@
 #include "asm/macroAssembler.hpp"
 #include "asm/macroAssembler.inline.hpp"
 #include "classfile/symbolTable.hpp"
+#include "code/aotCodeCache.hpp"
 #include "code/compiledIC.hpp"
 #include "code/debugInfoRec.hpp"
 #include "code/vtableStubs.hpp"
@@ -1391,7 +1392,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
     assert(vep_offset != -1,        "Must be set");
 #endif
 
-    __ flush();
+    // Code will be copied. No ICache sync required.
     nmethod* nm = nmethod::new_native_nmethod(method,
                                               compile_id,
                                               masm->code(),
@@ -1429,7 +1430,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
                          in_sig_bt,
                          in_regs);
     int frame_complete = ((intptr_t)__ pc()) - start;  // not complete, period
-    __ flush();
+    // Code will be copied. No ICache sync required.
     int stack_slots = SharedRuntime::out_preserve_stack_slots();  // no out slots at all, actually
     return nmethod::new_native_nmethod(method,
                                        compile_id,
@@ -1816,14 +1817,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
 
   Label safepoint_in_progress, safepoint_in_progress_done;
 
-  // Switch thread to "native transition" state before reading the synchronization state.
-  // This additional state is necessary because reading and testing the synchronization
-  // state is not atomic w.r.t. GC, as this scenario demonstrates:
-  //     Java thread A, in _thread_in_native state, loads _not_synchronized and is preempted.
-  //     VM thread changes sync state to synchronizing and suspends threads for GC.
-  //     Thread A is resumed to finish this native method, but doesn't block here since it
-  //     didn't see any synchronization is progress, and escapes.
-  __ mv(t0, _thread_in_native_trans);
+  __ mv(t0, _thread_in_vm);
 
   __ sw(t0, Address(xthread, JavaThread::thread_state_offset()));
 
@@ -2088,7 +2082,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
     }
   }
 
-  __ flush();
+  // Code will be copied. No ICache sync required.
 
   nmethod *nm = nmethod::new_native_nmethod(method,
                                             compile_id,
@@ -2123,6 +2117,12 @@ void SharedRuntime::generate_deopt_blob() {
   // Setup code generation tools
   int pad = 0;
   const char* name = SharedRuntime::stub_name(StubId::shared_deopt_id);
+  CodeBlob* blob = AOTCodeCache::load_code_blob(AOTCodeEntry::SharedBlob, BlobId::shared_deopt_id);
+  if (blob != nullptr) {
+    _deopt_blob = blob->as_deoptimization_blob();
+    return;
+  }
+
   CodeBuffer buffer(name, 2048 + pad, 1024);
   MacroAssembler* masm = new MacroAssembler(&buffer);
   int frame_size_in_words = -1;
@@ -2421,12 +2421,13 @@ void SharedRuntime::generate_deopt_blob() {
   // Jump to interpreter
   __ ret();
 
-  // Make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   _deopt_blob = DeoptimizationBlob::create(&buffer, oop_maps, 0, exception_offset, reexecute_offset, frame_size_in_words);
   assert(_deopt_blob != nullptr, "create deoptimization blob fail!");
   _deopt_blob->set_unpack_with_exception_in_tls_offset(exception_in_tls_offset);
+
+  AOTCodeCache::store_code_blob(*_deopt_blob, AOTCodeEntry::SharedBlob, BlobId::shared_deopt_id);
 }
 
 // Number of stack slots between incoming argument block and the start of
@@ -2453,13 +2454,18 @@ VMReg SharedRuntime::thread_register() {
 SafepointBlob* SharedRuntime::generate_handler_blob(StubId id, address call_ptr) {
   assert(is_polling_page_id(id), "expected a polling page stub id");
 
+  const char* name = SharedRuntime::stub_name(id);
+  CodeBlob* blob = AOTCodeCache::load_code_blob(AOTCodeEntry::SharedBlob, StubInfo::blob(id));
+  if (blob != nullptr) {
+    return blob->as_safepoint_blob();
+  }
+
   ResourceMark rm;
   OopMapSet *oop_maps = new OopMapSet();
   assert_cond(oop_maps != nullptr);
   OopMap* map = nullptr;
 
   // Allocate space for the code.  Setup code generation tools.
-  const char* name = SharedRuntime::stub_name(id);
   CodeBuffer buffer(name, 2048, 1024);
   MacroAssembler* masm = new MacroAssembler(&buffer);
   assert_cond(masm != nullptr);
@@ -2560,11 +2566,13 @@ SafepointBlob* SharedRuntime::generate_handler_blob(StubId id, address call_ptr)
   __ stop("Attempting to adjust pc to skip safepoint poll but the return point is not what we expected");
 #endif
 
-  // Make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   // Fill-out other meta info
-  return SafepointBlob::create(&buffer, oop_maps, frame_size_in_words);
+  SafepointBlob* sp_blob = SafepointBlob::create(&buffer, oop_maps, frame_size_in_words);
+
+  AOTCodeCache::store_code_blob(*sp_blob, AOTCodeEntry::SharedBlob, StubInfo::blob(id));
+  return sp_blob;
 }
 
 //
@@ -2579,10 +2587,15 @@ RuntimeStub* SharedRuntime::generate_resolve_blob(StubId id, address destination
   assert(StubRoutines::forward_exception_entry() != nullptr, "must be generated before");
   assert(is_resolve_id(id), "expected a resolve stub id");
 
+  const char* name = SharedRuntime::stub_name(id);
+  CodeBlob* blob = AOTCodeCache::load_code_blob(AOTCodeEntry::SharedBlob, StubInfo::blob(id));
+  if (blob != nullptr) {
+    return blob->as_runtime_stub();
+  }
+
   // allocate space for the code
   ResourceMark rm;
 
-  const char* name = SharedRuntime::stub_name(id);
   CodeBuffer buffer(name, 1000, 512);
   MacroAssembler* masm = new MacroAssembler(&buffer);
   assert_cond(masm != nullptr);
@@ -2648,12 +2661,13 @@ RuntimeStub* SharedRuntime::generate_resolve_blob(StubId id, address destination
   __ ld(x10, Address(xthread, Thread::pending_exception_offset()));
   __ far_jump(RuntimeAddress(StubRoutines::forward_exception_entry()));
 
-  // -------------
-  // make sure all code is generated
-  masm->flush();
+  // Code will be copied. No ICache sync required.
 
   // return the  blob
-  return RuntimeStub::new_runtime_stub(name, &buffer, frame_complete, frame_size_in_words, oop_maps, true);
+  RuntimeStub* rs_blob = RuntimeStub::new_runtime_stub(name, &buffer, frame_complete, frame_size_in_words, oop_maps, true);
+
+  AOTCodeCache::store_code_blob(*rs_blob, AOTCodeEntry::SharedBlob, StubInfo::blob(id));
+  return rs_blob;
 }
 
 // Continuation point for throwing of implicit exceptions that are
@@ -2698,6 +2712,11 @@ RuntimeStub* SharedRuntime::generate_throw_exception(StubId id, address runtime_
   const char* timer_msg = "SharedRuntime generate_throw_exception";
   TraceTime timer(timer_msg, TRACETIME_LOG(Info, startuptime));
 
+  CodeBlob* blob = AOTCodeCache::load_code_blob(AOTCodeEntry::SharedBlob, StubInfo::blob(id));
+  if (blob != nullptr) {
+    return blob->as_runtime_stub();
+  }
+
   CodeBuffer code(name, insts_size, locs_size);
   OopMapSet* oop_maps  = new OopMapSet();
   MacroAssembler* masm = new MacroAssembler(&code);
@@ -2756,6 +2775,8 @@ RuntimeStub* SharedRuntime::generate_throw_exception(StubId id, address runtime_
                                   (framesize >> (LogBytesPerWord - LogBytesPerInt)),
                                   oop_maps, false);
   assert(stub != nullptr, "create runtime stub fail!");
+
+  AOTCodeCache::store_code_blob(*stub, AOTCodeEntry::SharedBlob, StubInfo::blob(id));
   return stub;
 }
 

src/hotspot/cpu/riscv/stubRoutines_riscv.cpp

@@ -507,7 +507,17 @@ ATTRIBUTE_ALIGNED(4096) juint StubRoutines::riscv::_crc_table[] =
 };
 
 #if INCLUDE_CDS
-// nothing to do for riscv
+extern void StubGenerator_init_AOTAddressTable(GrowableArray<address>& external_addresses);
+
 void StubRoutines::init_AOTAddressTable() {
+  ResourceMark rm;
+  GrowableArray<address> external_addresses;
+  // publish static addresses referred to by riscv generator
+  // n.b. we have to use an extern call here because class
+  // StubGenerator, which provides the static method that knows how to
+  // add the relevant addresses, is declared in a source file rather
+  // than in a separately includeable header.
+  StubGenerator_init_AOTAddressTable(external_addresses);
+  AOTCodeCache::publish_external_addresses(external_addresses);
 }
 #endif // INCLUDE_CDS

src/hotspot/cpu/riscv/templateInterpreterGenerator_riscv.cpp

@@ -1213,7 +1213,7 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) {
   // Force all preceding writes to be observed prior to thread state change
   __ membar(MacroAssembler::LoadStore | MacroAssembler::StoreStore);
 
-  __ mv(t0, _thread_in_native_trans);
+  __ mv(t0, _thread_in_vm);
   __ sw(t0, Address(xthread, JavaThread::thread_state_offset()));
 
   // Force this write out before the read below

src/hotspot/cpu/riscv/upcallLinker_riscv.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2020, 2025, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2026, Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2020, 2023, Huawei Technologies Co., Ltd. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
@@ -330,7 +330,7 @@ address UpcallLinker::make_upcall_stub(jobject receiver, Symbol* signature,
 
   //////////////////////////////////////////////////////////////////////////////
 
-  __ flush();
+  // Code will be copied. No ICache sync required.
 
 #ifndef PRODUCT
   stringStream ss;

src/hotspot/cpu/riscv/vm_version_riscv.cpp

@@ -30,6 +30,7 @@
 #include "runtime/vm_version.hpp"
 #include "utilities/formatBuffer.hpp"
 #include "utilities/macros.hpp"
+#include "utilities/ostream.hpp"
 
 #include <ctype.h>
 
@@ -509,3 +510,62 @@ bool VM_Version::is_intrinsic_supported(vmIntrinsicID id) {
   }
   return true;
 }
+
+int VM_Version::cpu_features_size() {
+  return sizeof(RVExtFeatures);
+}
+
+void VM_Version::store_cpu_features(void* buf) {
+  memcpy(buf, RVExtFeatures::current(), sizeof(RVExtFeatures));
+}
+
+bool VM_Version::verify_aot_code_cache_features(void* features_buffer) {
+  RVExtFeatures* features_to_test = (RVExtFeatures*)features_buffer;
+  return RVExtFeatures::current()->verify_aot_code_cache_features(features_to_test);
+}
+
+// Print one feature using the same spelling as features_string(): single letter
+// extensions appear as "rvc"/"rvv" and multi-character extensions with a lower
+// case leading character ("Zba" -> "zba"). Must stay in sync with the feature
+// string built in VM_Version::setup_cpu_available_features().
+void VM_Version::print_feature_name(stringStream& ss, RVFeatureValue* feature) {
+  const char* pretty = feature->pretty();
+  if (strlen(pretty) == 1) {
+    ss.print("rv%s", pretty);
+  } else {
+    ss.print("%c%s", (char)tolower(pretty[0]), &pretty[1]);
+  }
+}
+
+void VM_Version::insert_features_names(RVExtFeatures* features, stringStream& ss) {
+  const char* sep = "";
+  int i = 0;
+  while (i < RVExtFeatures::MAX_CPU_FEATURE_INDEX) {
+    if (features->support_feature(i)) {
+      ss.print("%s", sep);
+      print_feature_name(ss, _feature_list[i]);
+      sep = ", ";
+    }
+    i += 1;
+  }
+}
+
+void VM_Version::get_cpu_features_name(void* features_buffer, stringStream& ss) {
+  RVExtFeatures* features = (RVExtFeatures*)features_buffer;
+  insert_features_names(features, ss);
+}
+
+void VM_Version::get_missing_features_name(void* features_set1, void* features_set2, stringStream& ss) {
+  RVExtFeatures* rv_ext_features_set1 = (RVExtFeatures*)features_set1;
+  RVExtFeatures* rv_ext_features_set2 = (RVExtFeatures*)features_set2;
+  const char* sep = "";
+  int i = 0;
+  while (i < RVExtFeatures::MAX_CPU_FEATURE_INDEX) {
+    if (rv_ext_features_set1->support_feature(i) && !rv_ext_features_set2->support_feature(i)) {
+      ss.print("%s", sep);
+      print_feature_name(ss, _feature_list[i]);
+      sep = ", ";
+    }
+    i += 1;
+  }
+}

src/hotspot/cpu/riscv/vm_version_riscv.hpp

@@ -36,6 +36,7 @@
 #include "utilities/sizes.hpp"
 
 class RiscvHwprobe;
+class stringStream;
 
 class VM_Version : public Abstract_VM_Version {
   friend RiscvHwprobe;
@@ -396,6 +397,15 @@ private:
       int idx = element_index(f);
       return (_features_bitmap[idx] & feature_bit(f)) != 0;
     }
+
+    bool verify_aot_code_cache_features(RVExtFeatures* features_to_test) const {
+      for (int i = 0; i < element_count(); i++) {
+        if (_features_bitmap[i] != features_to_test->_features_bitmap[i]) {
+          return false;
+        }
+      }
+      return true;
+    }
   };
 
   // enable extensions based on profile, current supported profiles:
@@ -523,6 +533,17 @@ private:
 
   // Check intrinsic support
   static bool is_intrinsic_supported(vmIntrinsicID id);
+
+  // AOT Code Cache support
+  static int  cpu_features_size();
+  static void store_cpu_features(void* buf);
+  static bool verify_aot_code_cache_features(void* features_buffer);
+  static void get_cpu_features_name(void* features_buffer, stringStream& ss);
+  static void get_missing_features_name(void* features_set1, void* features_set2, stringStream& ss);
+
+ private:
+  static void print_feature_name(stringStream& ss, RVFeatureValue* feature);
+  static void insert_features_names(RVExtFeatures* features, stringStream& ss);
 };
 
 #endif // CPU_RISCV_VM_VERSION_RISCV_HPP

src/hotspot/cpu/riscv/vtableStubs_riscv.cpp

@@ -137,7 +137,7 @@ VtableStub* VtableStubs::create_vtable_stub(int vtable_index, bool caller_is_c1)
   __ ld(t1, Address(xmethod, entry_offset));
   __ jr(t1);
 
-  masm->flush();
+  masm->invalidate_icache();
   bookkeeping(masm, tty, s, npe_addr, ame_addr, true, vtable_index, slop_bytes, 0);
 
   return s;
@@ -246,7 +246,7 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index, bool caller_is_c1)
   assert(SharedRuntime::get_handle_wrong_method_stub() != nullptr, "check initialization order");
   __ far_jump(RuntimeAddress(SharedRuntime::get_handle_wrong_method_stub()));
 
-  masm->flush();
+  masm->invalidate_icache();
   bookkeeping(masm, tty, s, npe_addr, ame_addr, false, itable_index, slop_bytes, 0);
 
   return s;

src/hotspot/cpu/s390/bytes_s390.hpp

@@ -1,63 +0,0 @@
-/*
- * Copyright (c) 2016, 2022, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2016, 2022 SAP SE. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#ifndef CPU_S390_BYTES_S390_HPP
-#define CPU_S390_BYTES_S390_HPP
-
-#include "memory/allStatic.hpp"
-
-class Bytes: AllStatic {
- public:
-  // Efficient reading and writing of unaligned unsigned data in
-  // platform-specific byte ordering.
-
-  // Use regular load and store for unaligned access.
-  //
-  // On z/Architecture, unaligned loads and stores are supported when using the
-  // "traditional" load (LH, L/LY, LG) and store (STH, ST/STY, STG) instructions.
-  // The penalty for unaligned access is just very few (two or three) ticks,
-  // plus another few (two or three) ticks if the access crosses a cache line boundary.
-  //
-  // In short, it makes no sense on z/Architecture to piecemeal get or put unaligned data.
-
-  static inline u2   get_native_u2(address p) { return *(u2*)p; }
-  static inline u4   get_native_u4(address p) { return *(u4*)p; }
-  static inline u8   get_native_u8(address p) { return *(u8*)p; }
-
-  static inline void put_native_u2(address p, u2 x) { *(u2*)p = x; }
-  static inline void put_native_u4(address p, u4 x) { *(u4*)p = x; }
-  static inline void put_native_u8(address p, u8 x) { *(u8*)p = x; }
-
-  // Efficient reading and writing of unaligned unsigned data in Java byte ordering (i.e. big-endian ordering)
-  static inline u2   get_Java_u2(address p) { return get_native_u2(p); }
-  static inline u4   get_Java_u4(address p) { return get_native_u4(p); }
-  static inline u8   get_Java_u8(address p) { return get_native_u8(p); }
-
-  static inline void put_Java_u2(address p, u2 x) { put_native_u2(p, x); }
-  static inline void put_Java_u4(address p, u4 x) { put_native_u4(p, x); }
-  static inline void put_Java_u8(address p, u8 x) { put_native_u8(p, x); }
-};
-
-#endif // CPU_S390_BYTES_S390_HPP

src/hotspot/cpu/s390/downcallLinker_s390.cpp

@@ -247,7 +247,7 @@ void DowncallLinker::StubGenerator::generate() {
 
   if (_needs_transition) {
     __ block_comment("thread_native2java {");
-    __ set_thread_state(_thread_in_native_trans);
+    __ set_thread_state(_thread_in_vm);
 
     if (!UseSystemMemoryBarrier) {
       __ z_fence(); // Order state change wrt. safepoint poll.
@@ -316,5 +316,5 @@ void DowncallLinker::StubGenerator::generate() {
 
   //////////////////////////////////////////////////////////////////////////////
 
-  __ flush();
+  // Code will be copied. No ICache sync required.
 }