This issue has been resolved

Oscar Users, Please see the details below on the Oscar outage.

Issue: New interactive jobs are slow to start Impact: Users may need to wait several minutes before an interactive job starts

OIT is investigating this issue. Please email support@ccv.brown.edu with any issues, questions or concerns.

Sincerely, CCV Research Technical Services

This issue has been resolved

Oscar Users, Please see the details below on the Oscar outage.

Issue: New jobs cannot be started on Oscar

Impact: Users cannot start new batch or interactive jobs Users cannot start new Open OnDemand sessions Users cannot check the status of existing jobs.

Instructions for users:

Wait for a message from our team before submitting a new job Please email support@ccv.brown.edu with any issues, questions or concerns.

Sincerely, CCV Research Technical Services

This issue has been resolved

There is an issue with the Open Ondemand server. New Open Ondeman (OOD) jobs cannot be launched. Old jobs are in the 'undetermined' status. We are working on the issue right now. Please email support@ccv.brown.edu with any issues, questions or concerns.

This issue has been resolved

Oscar Users,

Please see the details below on the Oscar outage.

Issue:

Oscar services are impacted by the current power issue at a data center. We are investigating the issue now and will update with more details soon.

Impact:

Users cannot access Oscar using these services

• ssh through ssh.ccv.brown.edu and sshcampus.ccv.brown.edu
• Remote IDE service for VSCode
• SMB mounts
• transfer server
• Login nodes
• Globus

Instructions for users:

Users can still connect to OOD from a web browser and launch interactive apps.

We will update this email thread with more information later.

Please email support@ccv.brown.edu with any issues, questions or concerns.

Sincerely,

CCV Research Technical Services

This issue has been resolved

Users are currently experiencing problems accessing Open OnDemand (ood.ccv.brown.edu) . Users are requested to use SSH to access Oscar.

This issue has been resolved

Oscar users, Please see the details below on the current outage:

Issue:

• Users cannot connect to Oscar due to a campus-wide outage. This outage is affecting multiple services.

Impact on Oscar:

• Users cannot connect to Oscar through SSH, Open OnDemand, File transfer services

• Oscar jobs should remain unaffected, unless they need to access the internet

We will update this email thread with more information later.

Please email support@ccv.brown.edu with any issues, questions or concerns.

Sincerely, CCV Research Technical Services

This issue has been resolved

Oscar Users,

Please see the details below on the current outage:

Issue:

Our engineers are investigating.

Impact:

Users will NOT be able to start an SSH session from these gateways:

• ssh.ccv.brown.edu

• sshcampus.ccv.brown.edu

• vscode1

Open OnDemand remains unaffected

All Slurm jobs are unaffected

The file system is unaffected

Instructions for Users:

Use Open OnDemand to access Oscar

We will update this email thread with more information later.

Please email support@ccv.brown.edu with any issues, questions or concerns.

Sincerely,

CCV Research Technical Services

This issue has been resolved

Oscar Users,

Please see the details below on GPU jobs were killed and requeued:

What is the issue: During the process of making configuration changes to the cluster, we have identified a scheduler bug: the scheduler kills and requeues GPU jobs when the scheduler configuration is updated.

What is the impact: A requeued GPU job will restart with the same job ID

Should you submit jobs? You can continue to submit jobs as usual, and they will proceed to run. We want to reassure you that before implementing any new configuration changes to the scheduler, we will provide a community-wide announcement to ensure that you have time to save your workloads.

Please email support@ccv.brown.edu with any issues, questions or concerns.

Sincerely, CCV Research Technical Services

This issue has been resolved

CCV will conduct the scheduled Oscar maintenance that upgrades the operating system and module system in January. To facilitate the transition, CCV has set up a mini cluster with the upgraded OS and module system for users to test their programs/jobs.

• Users can connect to the mini RHEL/9.2 cluster by running the command ssh -X login009 on the current login nodes (login007 or login008). For more details, refer to this page.
• Users can drop in during this Zoom meeting on Mondays and Wednesdays between 3 and 4pm before winter break for any assistance for the new cluster.
• We encourage users to test their programs/jobs on this cluster and reach out to us (support@ccv.brown.edu) for any help, issues or feedback.

Please see below for details on the scheduled maintenance.

Maintenance Window:

• Note: Below is the estimated window. The exact dates and times are still tentative.
• Start: 1/9/2024 5:00 am EST
• End: 1/12/2024 5:00 pm EST

Maintenance Description:

• The OS will be upgraded from RHEL/7.2 to RHEL/9.2. For exact details please refer to this documentation page
• We are increasing the cpu cores for priority-GPU accounts. Please see above link
• Oscar modules system will be migrated to LMod . For a comprehensive list of old and new modules refer to this documentation page.

Expected Impact:

• All Oscar services will be unavailable during the downtime
• Jobs which won’t complete by the beginning of the maintenance window won’t start and ‘myq/squeue’ will report (ReqNodeNotAvail, Reserved for maintenance)

Instructions for Users

• Users will need to resubmit jobs after the maintenance

• Users will need to update their job submission scripts since Oscar module names/versions will be different in the new module system.

• Locally installed packages/environments may not work after the maintenance due to OS upgrade. Users are recommended to reinstall and test their installed packages on the RHEL/9.2 mini cluster before the downtime.

This issue has been resolved

Oscar HPC experienced a service degradation with the VS Code server this evening. Our automated security systems identified a potential threat on the VS Code host node. Any ongoing sessions were terminated as a result. Our Systems team has identified that this host is safe and it should now be operational again.

This issue has been resolved

What’s was the issue: Route to Wide Area Network (WAN) is having an issue, resulting in no internet access to/from Oscar

Impact: Routing to the internet is broken Users are unable to connect to Oscar (ssh.ccv.brown.edu or ood.ccv.brown.edu) if not connected to Brown networks or VPN Jobs and Processes that need internet access such as - rsync, scp, git pull/push, file transfers, singularity pull etc. are not working at the moment and will fail. Users need to resubmit their failed jobs. Running and pending jobs that do not need external internet access are unaffected. Globus transfers are not working.

Current Status: Our engineers are currently working with the ISP to get this resolved as soon as possible.

Please email support@ccv.brown.edu with any issues, questions or concerns.

This issue has been resolved

Oscar Users, Please see below regarding issues on the Open OnDemand

What’s the issue: Open OnDemand (OOD) website is not accessible

Impact: Users are not able to access the OOD website/sessions Slurm jobs are NOT impacted All OOD app sessions are NOT impacted. They are just inaccessible right now.

Instructions for Users: Users are still able to access Oscar over SSH. Please wait until we send out another announcement to access the OOD website Users can check the status at status.ccv.brown.edu

We apologize for the inconvenience caused here. We are working on resolving this issue. Please reach out to us at support@ccv.brown.edu if you have any questions.

Sincerely, CCV Research Technical Services

This issue has been resolved

OIT Planned Maintenance: Oscar High Performance Computing

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled - Maintenance Window Start: 6/12/2023 5:00 am EST End: 6/19/2023 10:00 am EST

Maintenance Description During this maintenance period, we will be improving the electrical infrastructure for the entire data center. We will also be migrating Oscar’s underlying storage system from IBM’s GPFS to a new all-flash system provided by Vast.

Expected Impact All Oscar services will be unavailable during the downtime. Jobs which won’t complete by the beginning of the maintenance window won’t start and ‘myq/squeue’ will report. (ReqNodeNotAvail, Reserved for maintenance)

Users will need to resubmit jobs after the maintenance.

Any existing batch/interactive jobs, Open OnDemand sessions, Terminal, Screen, and Tmux sessions will be terminated.

Please email support@ccv.brown.edu with any issues, questions or concerns. We’ll be in touch again with additional instructions for the above changes (as appropriate).

This issue has been resolved

The Open OnDemand server (ood.ccv.brown.edu) is down. Users are not able to connect to OOD. We are actively working on the issue. Apologize for the inconvenience.

This issue has been resolved

The GPFS performance may experience a temporary decline as a result of a hardware malfunction on the storage server. The systems team is currently working on resolving the issue.

This issue has been resolved

Users are not able to transfer files using GUI applications - FileZilla, Cyberduck, and WinSCP. We will fix the issue as soon as possible, and post additional updates here.

This issue has been resolved

Several users have reported slowness in the file system while trying to open files on Oscar. We have contacted the vendors and we are still investigating this issue. We will post additional updates here.

This issue has been resolved

We are currently expecting the maintenance to be completed and services to be back online by January 12th at 10:00 am EST. Please note that this maintenance window is related to electrical work being performed. We will do our best to restore services as soon as possible.

This issue has been resolved

Oscar Users, Please the details below for the Oscar login nodes outage:

What’s the issue: The login nodes are currently very slow or unresponsive.

Impact: Users will not be able to connect to the login nodes through SSH This does NOT affect running jobs All apps other that the terminal app on Open OnDemand are working correctly

Alternatives: Use the OOD Desktop App or VNC Client to launch a desktop session and use the terminal inside that app to submit jobs, check job status, etc.

We are working on resolving this issue. We will send you an update as soon as the issue is resolved.

This issue has been resolved

Oscar users, Please see the details below for the transfer server outage:

What’s the issue: Due to an issue on the transfer server, users will not be able to access the transfer server (transfer.ccv.brown.edu) to perform file transfers, to and from Oscar. Users might be prompted for a password, but access will be denied to the server.

Alternatives to transfer server: Users can still transfer files using the gateway node (ssh.ccv.brown.edu) , SMB service, or Globus.

We are actively working on returning the transfer server to service. We will send out an update once the transfer server is back up and running.

This issue has been resolved

This is just a test

This issue has been resolved

Please see below regarding the file-system outage Oscar:

What’s the issue?

Due to an issue with the file-system:

• users will not be able to login
• All running/pending jobs are impacted

What’s the cause?

We have reopened the case with the vendor (IBM) and are actively working to identify the root cause and mitigate this issue as soon as possible.

Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV Research Technical Services

This issue has been resolved

What’s the issue?

Due to an issue with the file-system

• users will not be able to login
• All running/pending jobs are impacted

What’s the cause?

We are actively working with the vendor to identify the root cause. We will provide an update by noon tomorrow (April 26th).

Please email support@ccv.brown.edu with any issues, questions, or concerns.

This issue has been resolved

Oscar Users,

Please see below regarding the outage Oscar: What’s the issue?

Logins via SSH, VNC, VSCode etc are not working. Users might see the 'oscar2' prompt, but the current password will not be accepted.

What’s the cause?

We are still trying to identify the root cause and mitigate this. We apologize for the inconvenience. Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV Research Technical Services

This issue has been resolved

Please see below regarding the intermittent file-system slowness Oscar:

What’s the issue? Due to an intermittent slowness issue with the file-system:

• Users might notice slowness while logging in or doing tasks like opening files etc.
• Running jobs might perform slow due to I/O slowness
• VSCode, SSH, VNC sessions may take longer than usual to start

What’s the cause? We are working with the storage vendor to get this resolved. Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV Research Technical Services

This issue has been resolved

Maintenance Window: Start - 5:00 am, Tuesday, January 11, 2022 End - 4:00 pm, Friday, January 14, 2022 Please note: If unanticipated issues occur we expect to have them resolved by 8:00 am, Monday, January 17, 2022 at the latest

Maintenance Description: Performance upgrades to our GPFS storage cluster

Please email support@ccv.brown.edu with any issues, questions or concerns.

Sincerely, CCV Research Technical Services

This issue has been resolved

Please see below regarding the current license server disruption on Oscar:

What’s the issue?

• Commercial packages like (Matlab, Maple, ArcGIS) that require a license are currently inaccessible
• MATLAB GUI & Command Line prompt can’t be launched. Any running jobs using these modules may crash or fail
• This is due to license servers having issues

What’s the cause? The appropriate team is working to identify the root cause of this problem.

Please email support@ccv.brown.edu with any issues, questions or concerns.

This issue has been resolved

Oscar Users,

Please see below regarding the current partial outage on Oscar:

What’s the issue? Oscar is experiencing a partial outage, including:

1. Users will not be able to log-in via SSH, VNC, etc.
2. Running/pending jobs may have been impacted
3. Users will not be able to mount GPFS filesystem via SMB or CIFS

What’s the cause? We are working to mitigate the issue and return the cluster to service.

Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV User Services

This issue has been resolved

Please see below regarding the current outage on Oscar:

What’s the issue? Oscar is experiencing a complete outage, including: Users will not be able to log in via SSH, VNC, etc. Users will not be able to submit new jobs. Running jobs will NOT be impacted Users will not be able to mount GPFS filesystem via SMB or CIFS Globus, file-transfers are unavailable

What’s the cause? We are working to mitigate the issue and return the cluster to service.

Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV User Services

This issue has been resolved

Please see below regarding the current outage on Oscar What’s the issue? Oscar is experiencing issues with some SLURM related commands sbatch, sacct etc:

1. Users will not be able to submit new jobs via SBATCH
2. Users will not be able to query the SLURM database via sacct, myq, myjobinfo etc.
3. Running jobs are not impacted

What’s the cause?

We are working to mitigate the issue and return the cluster to service.

Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV User Services

This issue has been resolved

The following services are partially available at this time:

• Transfer nodes (accessible from internal Oscar network only). Not available from the public domain

The following services are not available at this time:

• SFTP & SSHFS to mount GPFS filesystem

Thank you for your continued patience.

CCV User Services

This issue has been resolved

Please see below regarding the current outage on Oscar

What’s the issue?

Oscar is experiencing a complete outage, including:

1. Users will not be able to log-in via SSH, VNC, etc.
2. All running jobs have been terminated
3. Users will not be able to mount GPFS filesystem via SMB or CIFS
4. Globus, file-transfers are unavailable.

What’s the cause? We are working to mitigate the issue and return the cluster to service.

Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV User Services

This issue has been resolved

Please see below regarding the current outage on Oscar

What’s the issue?

Oscar is experiencing complete outage, including:

1. Users will not be able to login via SSH, VNC etc.
2. All running jobs have been terminated
3. Users will not be able to mount GPFS filesystem via SMB or CIFS
4. Globus, file-transfers are unavailable.

What’s the cause? We are working to mitigate the issue and return the cluster to service.

Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV User Services

This issue has been resolved

Please see below regarding the current VNC outage on Oscar:

What’s the issue?

• Reconnecting to an existing VNC session will not work 
• New VNC sessions will not be able to start

What’s the cause? We are working to identify the root cause of this problem. Expect an update soon.

Please email support@ccv.brown.edu with any issues, questions, or concerns.

CCV User Services

This issue has been resolved

Please see below regarding the current MATLAB disruption on Oscar

What’s the issue?

• MATLAB is currently inaccessible from Oscar. Any running/scheduled MATLAB jobs will fail
• MATLAB GUI & Command Line prompt can’t be launched
• This is due to MATLAB license server having issues

What’s the cause?

• The appropriate team working to identify the root cause of this problem. Expect an update soon

Please email support@ccv.brown.edu with any issues, questions or concerns.

Sincerely, CCV User Services

This issue has been resolved

Oscar Users,

Please see below regarding the current outbound network outage on Oscar

What’s the issue?

• Oscar nodes cannot connect to outside networks and hosts
• This affects operations like wget, cloning git repos, any application calls that access the internet

What’s the cause?

• We are working to identify the root cause of this problem. Expect an update soon

Please email support@ccv.brown.edu with any issues, questions or concerns.

Sincerely, CCV User Services

This issue has been resolved

Oscar Users,

Please see below regarding the current Slurm runaway jobs on Oscar

What’s the issue?

• Slurm jobs after completion are not being cleared from the queue (runaway jobs). This means even if your previous job is finished it will still be shown as its in a RUNNING state to slurmctld
• This affects new jobs of users who have too many runaway jobs as their new jobs will be pending in the queue due to QOS limits

What’s the cause?

• We are working to identify the root cause of this problem. Expect an update soon

Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV User Services

This issue has been resolved

Please see below regarding the current GPFS file-system issue on Oscar.

What’s the issue?

• All OSCAR applications/modules, which reside in GPFS runtime are affected and users might see slowness in application startup times
• This also affects the Data storage pool. Any jobs writing directly to /gpfs/data will notice slow performance.

What’s the cause?

• We are working with the vendor (IBM) to determine the root cause of this issue and will send an update when we know more

Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV User Services

This issue has been resolved

Please see below regarding the current network issue on Oscar.

What’s the issue? This affects operations like wget, cloning git repos, any application calls that access the internet. Users can still login to Oscar and any running or pending jobs are not affected.

What’s the cause? One of the gateway servers suffered a hardware failure. Outbound Internet traffic from Oscar is unavailable. We are working to determine the root cause of this issue and will send an update when we know more

Please email support@ccv.brown.edu with any issues, questions, or concerns.

Sincerely, CCV User Services

This issue has been resolved

Please click here for additional details regarding the upcoming CCV maintenance work:

This issue has been resolved

The Oscar cluster is inaccessible from remote sites. Admins are looking into it. We will update soon.

This issue has been resolved

The file-system performance is degraded, some users might experience slowness while creating virtual-environments or performing small file IO operations. We are investigating the issue.

This issue has been resolved

Who is impacted:

• SMB users who access OSCAR locally via the network drive on Windows, Finder on Macs, or Linux

What services are impacted:

• SMB service
• Note: the secondary mount oscarcifs.ccv.brown.edu will no longer be available as of September 1, 2020. Please begin utilizing the SMB service as soon as possible

This issue has been resolved

Whats the issue? SLURM daemons (sbatch, squeue, sinfo, scancel, srun etc.) are slow to respond. A new job submission will also be slow. The new VNC sessions are unable to start.

What's the cause? A user submitted over 30 thousand jobs overloading SLURM daemons.

How will it be resolved? SLURM is canceling these jobs. We have also implemented the MaxJobPerUserLimit.

When will it be resolved? Once the jobs are fully cancelled, SLRUM should return back to normal (in the coming hours)

This issue has been resolved

Monday July 13, 2020 The Oscar cluster began experiencing issues with slow I/O performance with the new GPFS file system. In order to help mitigate this issue, we are moving some components to a faster pool of disks.

We appreciate your patience as we troubleshoot this issue. We will send another update once it is resolved. Please email [support@ccv.brown.edu] with any questions.

This issue has been resolved

The CCV OSCAR System is down for filesystem maintenance during the week of June 22.

This issue has been resolved

Slow filesystem responsiveness, operations such as running shell commands, starting applications, data transfers, etc are affected.

This issue has been resolved

Users could not log into Oscar. The login004 node did not respond for ssh. Admins are investigating the issue.

This issue has been resolved

The Oscar cluster began experiencing issues with the home filesystem around 5:00am. Admins are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed.